How to Build a Medallion Architecture in Databricks with PySpark

Raw data rarely arrives ready for analysis. Files appear late, schemas change without notice, identifiers go missing, and the same event may be delivered more than once. If every dashboard and machine learning workflow handles those problems independently, inconsistent results are almost guaranteed.

A Medallion Architecture creates clear boundaries between raw, validated, and business-ready data. In this tutorial, we will build those Bronze, Silver, and Gold layers in Databricks using a small commerce-events dataset. We will also cover the less visible parts of the design: rejected records, schema drift, duplicate handling, late data, orchestration, and recovery.

What is a Medallion Architecture in Databricks?

A Medallion Architecture is a lakehouse design pattern that improves data progressively as it moves through multiple layers. Databricks describes the pattern as a recommended practice rather than a requirement for every workload.

The three standard layers are:

Layer	Purpose	Typical work	Quality promise	Main consumers
Bronze	Preserve source data	Incremental ingestion and metadata capture	Data remains close to what the source delivered	Silver pipelines, audit, and recovery processes
Silver	Create trusted records	Parsing, validation, deduplication, normalization, and joins	Records follow documented technical rules	Gold pipelines, data engineers, analysts, and data scientists
Gold	Publish useful data products	Business rules, dimensional models, and aggregations	Tables answer defined business questions	BI tools, applications, ML workloads, and decision-makers

The names alone do not create the architecture. Each layer needs a clear contract describing what it accepts, what it changes, what it guarantees, and how it handles failure.

Databricks provides a useful overview of the Medallion Lakehouse Architecture, including the intended responsibilities and consumers of each layer.

The pipeline we will build

Assume that an application exports commerce events as JSON files. Each event may contain these fields:

{
  "event_id": "evt-1048",
  "customer_id": "cus-392",
  "event_type": "purchase",
  "event_timestamp": "2026-09-10T13:42:18Z",
  "source_updated_at": "2026-09-10T13:43:02Z",
  "product_id": "sku-271",
  "quantity": 2,
  "unit_price": 39.50,
  "currency": "USD"
}

Real input will not always be this clean. Our pipeline must also account for duplicate IDs, invalid timestamps, missing customer IDs, unexpected fields, late events, and corrected versions of earlier events.

The data path will be:

JSON files
  -> Bronze raw events
  -> Silver valid events + Silver quarantine
  -> Gold daily commerce metrics

The examples assume that Unity Catalog is enabled and that your workspace can read the source storage location. Replace the catalog name, paths, permissions, and deployment settings with values appropriate for your environment.

Step 1: Organize the layers in Unity Catalog

Unity Catalog identifies data objects with a three-level namespace: catalog.schema.table. For this tutorial, we will use one domain catalog with a schema for each Medallion layer:

commerce.bronze.events_raw
commerce.silver.events_valid
commerce.silver.events_quarantine
commerce.gold.daily_commerce_metrics

Create the catalog and schemas:

CREATE CATALOG IF NOT EXISTS commerce;

CREATE SCHEMA IF NOT EXISTS commerce.bronze
COMMENT 'Raw commerce data retained for replay and audit';

CREATE SCHEMA IF NOT EXISTS commerce.silver
COMMENT 'Validated and conformed commerce records';

CREATE SCHEMA IF NOT EXISTS commerce.gold
COMMENT 'Business-facing commerce data products';

This layout is easy to follow, but it is not the only correct model. Some organizations separate development and production with catalogs, while others organize catalogs by business domain. Choose boundaries according to ownership, access, environment isolation, and managed-storage requirements—not simply because an example uses three schemas.

Permissions should also reflect how the layers are consumed. Most analysts should not need direct access to raw Bronze records, while Gold datasets generally need broader read access and more stable contracts.

Step 2: Build the Bronze layer with Auto Loader

The Bronze layer should preserve source fidelity. It may add operational metadata, but it should not calculate revenue, standardize business definitions, or silently discard malformed records.

Databricks Auto Loader incrementally discovers and processes new files from cloud object storage. The following PySpark example reads JSON files and writes them to a Bronze Delta table:

from pyspark.sql import functions as F

source_path = "s3://example-bucket/commerce/events/"
schema_path = "s3://example-bucket/checkpoints/events_schema/"
checkpoint_path = "s3://example-bucket/checkpoints/events_bronze/"

bronze_events = (
    spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        .option("cloudFiles.schemaLocation", schema_path)
        .option("cloudFiles.schemaEvolutionMode", "rescue")
        .option("rescuedDataColumn", "_rescued_data")
        .load(source_path)
        .withColumn("_ingested_at", F.current_timestamp())
        .withColumn("_source_file", F.col("_metadata.file_path"))
)

(
    bronze_events.writeStream
        .option("checkpointLocation", checkpoint_path)
        .trigger(availableNow=True)
        .toTable("commerce.bronze.events_raw")
)

The availableNow trigger processes all currently available data and then stops. This is useful when a workflow should run incrementally on a schedule instead of keeping a stream active continuously.

The ingestion and source-file columns make it possible to trace a record back to its arrival and origin. They are important when investigating duplicates, replaying a particular batch, or explaining why a downstream metric changed.

What happens when the source schema changes?

Schema change is an operational decision, not just a convenience setting. In this example, Auto Loader uses rescue mode, which records fields that do not match the known schema in _rescued_data instead of adding them automatically or losing them silently.

Other workloads may allow compatible new columns to evolve the schema. Whichever option you select, document what happens when a column is added, removed, renamed, or sent with a different type. Databricks explains the available behaviors in its Auto Loader schema evolution documentation.

Bronze should preserve problematic input so that it can be inspected and reprocessed. It should not promise that every field is valid or ready for analysis.

Step 3: Create the Silver validation contract

Silver is where raw events become dependable technical records. Before writing transformation code, define the rules the table will enforce.

For this dataset:

• event_id must be present.
• customer_id must be present.
• event_timestamp must parse successfully.
• event_type must be product_view, add_to_cart, purchase, or refund.
• Quantity and unit price cannot be negative.
• Currency values are normalized to uppercase.
• Invalid records are quarantined with a reason rather than discarded.

The following transformation applies those rules:

from pyspark.sql import functions as F

allowed_events = ["product_view", "add_to_cart", "purchase", "refund"]

prepared_events = (
    spark.table("commerce.bronze.events_raw")
        .withColumn("event_type", F.lower(F.trim("event_type")))
        .withColumn("currency", F.upper(F.trim("currency")))
        .withColumn("event_ts", F.to_timestamp("event_timestamp"))
        .withColumn("source_updated_ts", F.to_timestamp("source_updated_at"))
        .withColumn("quantity", F.col("quantity").cast("long"))
        .withColumn("unit_price", F.col("unit_price").cast("decimal(18,2)"))
        .withColumn(
            "validation_reason",
            F.when(F.col("event_id").isNull(), "missing_event_id")
             .when(F.col("customer_id").isNull(), "missing_customer_id")
             .when(F.col("event_ts").isNull(), "invalid_event_timestamp")
             .when(~F.col("event_type").isin(allowed_events), "invalid_event_type")
             .when(F.col("quantity") < 0, "negative_quantity")
             .when(F.col("unit_price") < 0, "negative_unit_price")
        )
)

valid_events = prepared_events.filter(F.col("validation_reason").isNull())
quarantined_events = prepared_events.filter(F.col("validation_reason").isNotNull())

A single validation-reason column keeps the example readable. A production design may retain multiple failed rules, severity levels, source ownership, and remediation status.

The quarantine table is not a substitute for monitoring. Teams still need alerts, ownership, and a process for deciding whether a rejected record should be corrected, replayed, or deliberately excluded.

Step 4: Handle duplicates, corrections, and late events

These problems are related, but they are not identical:

• A duplicate delivery repeats the same event.
• A correction changes an event that was already received.
• A late event is valid but arrives after its business timestamp.

Running MERGE does not automatically decide which version is correct. First, establish a deterministic selection rule. Here, we keep the latest source update for each event_id, falling back to ingestion time when necessary:

from delta.tables import DeltaTable
from pyspark.sql import Window
from pyspark.sql import functions as F

latest_event_window = (
    Window.partitionBy("event_id")
          .orderBy(
              F.col("source_updated_ts").desc_nulls_last(),
              F.col("_ingested_at").desc()
          )
)

latest_valid_events = (
    valid_events
        .withColumn("version_rank", F.row_number().over(latest_event_window))
        .filter(F.col("version_rank") == 1)
        .drop("version_rank", "validation_reason")
)

silver_table_name = "commerce.silver.events_valid"

if not spark.catalog.tableExists(silver_table_name):
    latest_valid_events.write.format("delta").saveAsTable(silver_table_name)
else:
    silver_table = DeltaTable.forName(spark, silver_table_name)

    (
        silver_table.alias("target")
            .merge(
                latest_valid_events.alias("source"),
                "target.event_id = source.event_id"
            )
            .whenMatchedUpdateAll(
                condition="source.source_updated_ts >= target.source_updated_ts"
            )
            .whenNotMatchedInsertAll()
            .execute()
    )

Your real ordering rule must follow the semantics of the source system. If source_updated_at is unreliable, another version number or change sequence may be required.

Late-arriving records should retain both event time and ingestion time. Event time controls the business date; ingestion time explains when the platform learned about the event. When a late purchase is merged into Silver, the affected Gold period must be recalculated so the business summary does not remain stale.

The quarantine write should also be idempotent. A production pipeline can generate a stable rejection identifier from the source file, record position, event ID, and failed rule, then merge it into commerce.silver.events_quarantine. A simple append may create repeated rejection records whenever a historical batch is replayed.

Step 5: Build the Gold daily commerce table

Gold tables should be designed around a named consumer and question. Our example answers: What were daily commerce activity and net revenue?

from pyspark.sql import functions as F

silver_events = spark.table("commerce.silver.events_valid")

daily_metrics = (
    silver_events
        .withColumn("activity_date", F.to_date("event_ts"))
        .groupBy("activity_date", "currency")
        .agg(
            F.count("*").alias("total_events"),
            F.countDistinct("customer_id").alias("unique_customers"),
            F.sum(
                F.when(F.col("event_type") == "purchase", 1).otherwise(0)
            ).alias("purchase_events"),
            F.sum(
                F.when(
                    F.col("event_type") == "purchase",
                    F.col("quantity") * F.col("unit_price")
                ).otherwise(F.lit(0))
            ).alias("gross_revenue"),
            F.sum(
                F.when(
                    F.col("event_type") == "refund",
                    F.col("quantity") * F.col("unit_price")
                ).otherwise(F.lit(0))
            ).alias("refunded_revenue")
        )
        .withColumn(
            "net_revenue",
            F.col("gross_revenue") - F.col("refunded_revenue")
        )
)

(
    daily_metrics.write
        .format("delta")
        .mode("overwrite")
        .saveAsTable("commerce.gold.daily_commerce_metrics")
)

The conditional sums matter. count(event_type == "purchase") would count non-null Boolean results, including false, rather than only purchase rows. A conditional sum, as shown above, makes the intended calculation explicit.

The full overwrite keeps this small tutorial easy to follow. For a large production table, recompute only the dates affected by new or corrected Silver events and replace or merge those Gold records. Select any lookback window from observed source lateness, correction behavior, reporting requirements, and cost. For unbounded corrections, consider change data capture or another targeted strategy instead of assuming a fixed window is sufficient.

Gold does not have to mean one universal table. Finance, merchandising, customer analytics, and operational teams may need separate Gold products with different definitions, access controls, refresh schedules, and service-level expectations.

Step 6: Orchestrate the dependencies

The pipeline has a simple dependency graph:

Bronze ingestion
  -> Silver validation and merge
  -> Gold aggregation
  -> publication checks

Databricks offers two relevant orchestration approaches. Lakeflow Spark Declarative Pipelines can manage declarative batch and streaming flows, incremental processing, and data-quality expectations. Lakeflow Jobs can orchestrate notebooks, pipeline tasks, SQL tasks, and other workload types.

Whichever approach you use, define measurable checks at every boundary:

• Is Bronze receiving data within the expected freshness window?
• Did the amount of rescued data change unexpectedly?
• Which Silver validation rules are rejecting records?
• Does Silver contain more than one accepted record per event ID?
• Were late changes propagated to the affected Gold dates?
• Is the Gold table current and internally consistent?

An expectation can warn, drop a record, or fail a pipeline update. The correct action depends on business impact. A missing optional product attribute may justify a warning, while a broken primary key in a financial dataset may need to stop publication.

Step 7: Optimize and monitor the tables

Do not partition every Gold table by date automatically. Static partitioning can create small files, skew, and rigid layout boundaries when the table or access pattern does not justify it.

For new Delta tables, current Databricks guidance generally favors liquid clustering over defaulting to static partitions or ZORDER. Choose clustering keys from actual filters and joins, and confirm that the table is large enough to benefit. Predictive optimization may also automate maintenance for eligible Unity Catalog managed tables.

Operational monitoring should cover more than job success. Track:

• Source and table freshness
• Input and output row counts
• Rescued and quarantined records
• Duplicate and correction frequency
• Pipeline duration and retry behavior
• Gold reconciliation checks
• Query performance and compute usage

Use Delta table history and Lakeflow pipeline event logs when investigating a change. Alerts should identify the affected dataset, failed rule, pipeline run, and responsible owner instead of reporting only that “a job failed.”

How to recover and reprocess data safely

The layered architecture gives teams deliberate recovery points. If Silver transformation logic is wrong, correct the code and rebuild the affected Silver range from Bronze. If a Gold calculation changes, rebuild Gold from trusted Silver records without pulling the original data again.

Checkpoints represent streaming progress, so they should not be deleted casually. Resume an existing stream with its existing checkpoint. Use a new checkpoint only as part of a controlled replay, and make sure the target write is idempotent so the replay does not multiply records.

Retention policies must reflect replay, audit, regulatory, storage, and source-availability requirements. “Keep Bronze for 90 days” is not a universal rule. If the source cannot reproduce historical records, deleting Bronze may also remove your only reliable recovery path.

Common Medallion Architecture mistakes

Applying business logic in Bronze

Bronze should retain source fidelity. Calculating KPIs or overwriting raw values at ingestion makes it harder to replay data after business rules change.

Writing ingestion directly into Silver

This collapses raw capture and validation into one failure boundary. A corrupt record or unexpected schema change can prevent you from retaining what the source actually sent.

Dropping invalid records without a quarantine path

Removing bad data can make a pipeline appear healthier than it is. Preserve the record, failed rule, source metadata, and remediation status where the use case requires investigation.

Treating ingestion time as event time

Late events belong to the period in which the activity happened, not necessarily the day the platform received them. Keep both timestamps and define how downstream periods are refreshed.

Using MERGE without resolving source duplicates

Multiple source rows for one target key create ambiguity. Select an accepted source version deterministically before performing the merge.

Building Gold tables without named consumers

A collection of unexplained aggregates becomes another data swamp. Every Gold product should have an owner, definition, consumer, refresh expectation, and quality contract.

Optimizing before measuring

Partitioning, clustering, caching, and materialization all have trade-offs. Start with table size and query behavior, then select the technique that addresses the observed problem.

Calling three folders an architecture

Physical separation is useful, but governance, lineage, validation, ownership, recovery, and operational controls are what make the layers dependable.

When you may not need all three layers

Three separately materialized layers can add storage, compute, latency, and maintenance. A small proof of concept or a stable internal dataset with one low-risk consumer may not require every layer as a separate table.

That does not mean validation should disappear. A team might retain a replayable landing area and combine some Silver and Gold processing, or publish a view rather than another materialized copy. The simplification should be intentional, documented, and consistent with recovery and quality requirements.

Use the layers because each one creates a valuable boundary—not because the names have become standard.

Implementation checklist

Before moving a Medallion pipeline into production, confirm that:

• Each layer has a documented owner and contract.
• Business keys and record-version rules are defined.
• Bronze can support the required replay window.
• Schema-evolution behavior is deliberate and monitored.
• Invalid records have an observable failure path.
• Deduplication and correction rules are deterministic.
• Event-time and ingestion-time behavior are documented.
• Silver-to-Gold definitions are agreed with consumers.
• Unity Catalog permissions are tested for each user group.
• Data-quality expectations have appropriate actions.
• Pipeline failure and freshness alerts reach an owner.
• Backfill and recovery procedures have been tested.
• Every Gold table has a named consumer and purpose.

Final perspective

A Medallion Architecture works when every layer makes the data more dependable and useful. Bronze preserves what arrived, Silver establishes trustworthy records, and Gold publishes stable data products for a defined business purpose. The hard part is not creating three schemas; it is designing the contracts, failure paths, update rules, and operational controls between them.

Teams that need additional capacity to design, implement, or stabilize these pipelines can evaluate specialized Databricks data engineering services alongside their internal platform capabilities. The right support should still begin with the workload, consumers, governance needs, and recovery requirements—not a predefined layer template.