Modern enterprises generate data from everywhere:
- Operational databases
- SaaS applications
- APIs
- Event streams
- IoT devices
- Application logs
- CRM and ERP systems
- Documents and unstructured content
- Machine learning systems
- Customer interactions
The challenge is no longer simply storing data.
The real architectural challenge is:
How do we transform raw, heterogeneous, continuously changing data into trustworthy, governed, reusable data products that can power analytics, machine learning, and AI applications?
This is where the Medallion Architecture becomes extremely useful.
The Medallion Architecture is a logical data design pattern that progressively improves the quality and usability of data as it moves through Bronze → Silver → Gold layers. Databricks describes it as a multi-layered approach for building reliable lakehouse data products.
But there is a common misconception:
Medallion Architecture is not simply "put raw data in Bronze, clean it in Silver, aggregate it in Gold."
For an architect, the more important questions are:
- What belongs in each layer?
- Where should data quality be enforced?
- Where should CDC be processed?
- Where should business rules live?
- How should domains own their data?
- How should governance work?
- Should every dataset have all three layers?
- How should streaming and batch coexist?
- How should ML and GenAI consume these datasets?
- How do we design for replayability and auditability?
- How do we prevent Bronze/Silver/Gold from becoming another set of data silos?
This article explores those questions.
1. What Is Medallion Architecture?
Medallion Architecture is a logical layering pattern for organizing data according to its level of refinement.
The classic model is:
SOURCE SYSTEMS
│
┌──────────────┼──────────────┐
│ │ │
DB APIs Events
│ │ │
└──────────────┼──────────────┘
│
▼
┌───────────┐
│ BRONZE │
│ RAW │
└─────┬─────┘
│
Clean / Validate
│
▼
┌───────────┐
│ SILVER │
│ CURATED │
└─────┬─────┘
│
Business Modeling
│
▼
┌───────────┐
│ GOLD │
│ BUSINESS │
│ READY │
└─────┬─────┘
│
┌────────────┼─────────────┐
▼ ▼ ▼
BI ML AI
Dashboards Models Agents
Databricks defines Bronze as raw data, Silver as validated/refined data, and Gold as enriched/business-oriented data.
The fundamental principle is:
Data quality and semantic value increase as data moves through the layers.
2. Why Was This Architecture Needed?
Traditional enterprise data platforms often looked like this:
Applications
│
├── ETL ──→ Data Warehouse
│
├── ETL ──→ Reporting Database
│
├── ETL ──→ ML Platform
│
└── ETL ──→ Data Lake
Over time, this can create:
- Duplicate pipelines
- Duplicate datasets
- Conflicting business definitions
- Difficult lineage
- Expensive storage
- Inconsistent metrics
- Multiple versions of "customer"
- Difficult reprocessing
- Poor governance
The lakehouse approach attempts to provide a common foundation for analytics, BI, ML, and AI workloads. Databricks describes the lakehouse as combining characteristics of data lakes and data warehouses while supporting multiple workloads over shared data.
Medallion Architecture provides a structured way to progressively transform that data.
3. Bronze Layer — The System of Record
Bronze is the raw ingestion layer.
Its primary responsibility is:
Preserve what arrived from the source.
Not:
"Make the data beautiful."
That distinction is extremely important.
Databricks recommends preserving the raw state of source data in Bronze so downstream layers can be rebuilt when necessary. Bronze can receive data through batch or streaming mechanisms and from sources such as cloud storage, Kafka, and federated systems.
3.1 What Goes Into Bronze?
Examples:
PostgreSQL
MySQL
Oracle
SAP
Salesforce
Kafka
Kinesis
REST APIs
IoT
Application Logs
Cloud Storage
CDC Streams
A healthcare enterprise might have:
EHR
│
▼
Bronze
Claims
│
▼
Bronze
Patient Events
│
▼
Bronze
Provider Directory
│
▼
Bronze
Call Center Events
│
▼
Bronze
The Bronze layer captures these source-specific datasets.
4. What Should Bronze Look Like?
Suppose the source produces:
{
"customerId": "C123",
"name": "John Smith",
"email": "john@example.com",
"amount": "125.50",
"timestamp": "2026-08-18T08:00:00Z"
}
The Bronze layer should generally preserve the source structure rather than immediately applying extensive business transformations.
You may add ingestion metadata such as:
source_system
ingestion_timestamp
batch_id
file_name
event_timestamp
source_partition
record_hash
Databricks specifically recommends retaining source fields for auditability and allowing flexible handling of unexpected schemas in Bronze.
5. Bronze Is Not Your Reporting Layer
One of the biggest architectural mistakes is allowing analysts and applications to directly depend on Bronze.
Bronze is typically:
- Raw
- Source-oriented
- High volume
- Potentially inconsistent
- Not fully validated
- Subject to source-specific structures
For example:
customer_name
CustomerName
cust_nm
customerName
Different source systems may represent the same concept differently.
Bronze preserves those differences.
Silver resolves them.
6. Bronze Should Enable Replayability
This is one of the most important architectural benefits.
Imagine your Silver transformation contains a bug.
Without raw historical data:
Source
↓
Transformation bug
↓
Incorrect Silver
↓
Incorrect Gold
You may have to go back to the source system.
With a durable Bronze layer:
Bronze
│
┌───────┴────────┐
▼ ▼
Silver v1 Silver v2
│
▼
Gold
You can rebuild downstream layers.
That makes Bronze an important recovery and replay boundary.
Databricks explicitly highlights the ability to rebuild downstream layers from Bronze as a key characteristic.
7. Should Bronze Be Immutable?
Conceptually, Bronze should preserve the source history.
However, "immutable" does not mean you can never technically perform maintenance operations on the physical table.
The architectural principle is:
Do not destroy source fidelity merely to make downstream processing convenient.
If the source produces:
UPDATE Customer
SET status = 'ACTIVE'
you need to decide whether the Bronze representation should preserve:
Before:
status = INACTIVE
After:
status = ACTIVE
or whether the ingestion mechanism captures only the current state.
For CDC-heavy systems, preserving change events can be extremely valuable.
8. CDC and the Bronze Layer
Consider an operational database:
Customer
---------
id
name
status
updated_at
A CDC stream might produce:
INSERT
UPDATE
UPDATE
DELETE
Bronze can preserve these events.
Conceptually:
Operational DB
│
▼
CDC
│
▼
Bronze
│
▼
Silver
│
▼
Current Customer State
Silver can then construct a clean current-state representation or a historical representation depending on downstream requirements.
This separation is powerful because the raw change history remains available.
9. Silver Layer — Where Data Becomes Trustworthy
If Bronze is about preservation, Silver is about trust.
The Silver layer typically handles:
- Schema enforcement
- Data validation
- Deduplication
- Type casting
- Null handling
- Standardization
- Joining
- Enrichment
- Late-arriving data
- Out-of-order events
- CDC processing
- Business-level cleansing
Databricks describes Silver as the layer where cleansing, validation, deduplication, normalization, joins, schema evolution, and other refinement activities occur.
10. Example: Bronze → Silver
Imagine Bronze contains:
customer_id = "001"
customer_name = " JOHN SMITH "
email = "JOHN@EXAMPLE.COM"
age = "35"
country = "US"
Silver might produce:
customer_id = 1
customer_name = "John Smith"
email = "john@example.com"
age = 35
country_code = "US"
Now the data has:
- Correct types
- Standardized values
- Validated fields
- Consistent naming
Silver becomes much more useful to downstream consumers.
11. Data Quality Belongs Heavily in Silver
A mature architecture shouldn't treat data quality as a single validation job.
Think of quality as a progressive process:
Bronze
│
│ Basic ingestion validation
▼
Silver
│
│ Strong structural + semantic validation
▼
Gold
│
│ Business KPI validation
▼
Consumers
Databricks recommends applying data quality checks across the medallion layers, and Lakeflow pipelines support expectations that can validate records and either fail updates, drop invalid records, or track quality metrics depending on configuration.
12. Data Quality Example
Suppose:
customer_id IS NOT NULL
email IS VALID
age >= 0
country_code IN supported values
transaction_amount >= 0
A pipeline could classify records as:
Incoming Records
│
┌─────────┴─────────┐
▼ ▼
Valid Invalid
│ │
▼ ▼
Silver Quarantine
The important architectural point is:
Don't silently discard bad data.
Invalid records may be operationally important.
You may need:
- Quarantine tables
- Error reason
- Source information
- Processing timestamp
- Pipeline version
- Original record
This enables remediation and audit.
13. Silver Is Often the Most Important Layer
For architects, Silver is arguably the most strategically important layer.
Why?
Because Gold is usually purpose-specific.
Silver is reusable.
For example:
Silver
│
┌───────────────┼────────────────┐
▼ ▼ ▼
BI ML AI
│ │ │
▼ ▼ ▼
Gold Sales Feature Sets RAG Data
A well-designed Silver layer becomes a shared enterprise data foundation.
14. Silver and Canonical Data Models
Suppose five systems represent customers differently.
CRM
E-Commerce
Billing
Support
Marketing
You don't want every downstream consumer to understand five definitions.
Instead:
Source Systems
│
▼
Bronze
│
▼
Canonical Silver
│
┌────────────┼────────────┐
▼ ▼ ▼
Sales ML AI
Silver becomes the place where the organization starts establishing common semantics.
For example:
Customer
CustomerAccount
Product
Order
Transaction
Provider
Patient
Claim
Interaction
These entities become standardized building blocks.
15. Gold Layer — Business-Ready Data
Gold is where data becomes optimized for specific business and analytical use cases.
Typical Gold datasets include:
daily_sales
customer_lifetime_value
revenue_by_region
customer_churn_metrics
executive_kpis
product_performance
provider_performance
Databricks describes Gold as the layer containing highly refined datasets aligned with business functions, often aggregated and optimized for analytics and reporting.
16. Gold Is Not Simply "Aggregated Data"
This is another common misconception.
Gold can contain:
- Aggregates
- Dimensional models
- Business metrics
- Data marts
- Feature datasets
- Application-oriented datasets
- Domain-specific data products
The defining characteristic is:
Gold is optimized for a specific consumer or business purpose.
17. Example: Sales Domain
Suppose Silver contains:
customers
orders
order_items
products
payments
Gold could expose:
gold.sales_daily
gold.customer_lifetime_value
gold.product_performance
gold.regional_revenue
A BI dashboard shouldn't need to join 15 Silver tables every time it loads.
Gold can provide a semantic layer optimized for the business question.
18. Gold for Machine Learning
Gold isn't limited to BI.
ML workloads can consume refined datasets from Silver and Gold.
For example:
Silver
├── customer_transactions
├── customer_interactions
├── product_views
└── support_events
│
▼
Feature
Engineering
│
▼
Gold
│
▼
ML Training
However, architects should avoid automatically forcing all ML features into Gold.
Feature engineering may require:
- Point-in-time correctness
- Historical state
- High-frequency events
- Specialized feature stores
- Online/offline serving
The correct architecture depends on the ML use case.
19. Gold for Generative AI
This becomes particularly interesting in modern AI architectures.
Consider an enterprise RAG system.
Raw documents may arrive as:
PDF
DOCX
HTML
Email
Knowledge Base
CRM
Ticketing System
Bronze:
Raw documents
Raw metadata
Raw ingestion events
Silver:
Parsed documents
Clean text
Normalized metadata
Access-control metadata
Document versions
Chunks
Gold:
AI-ready knowledge assets
Retrieval metadata
Business entities
Semantic relationships
Curated knowledge views
Then:
Gold / Curated Knowledge
│
▼
Embeddings
│
▼
Vector Database
│
▼
RAG
│
▼
Agentic AI System
This is where Medallion Architecture becomes particularly powerful for enterprise AI architecture.
20. Medallion Architecture + Agentic AI
Imagine an enterprise support agent.
The agent needs:
Customer Profile
Order History
Support Tickets
Product Documentation
Policies
Entitlements
Usage Data
Instead of letting every agent query raw operational systems independently:
Agent
├── CRM API
├── ERP API
├── Ticket API
├── Database
├── File System
└── Knowledge Base
we can curate reusable data products:
Enterprise Data Platform
│
Silver
│
┌────────────────┼────────────────┐
▼ ▼ ▼
Customer Orders Support
│ │ │
└────────────────┼────────────────┘
▼
AI Data Products
│
▼
Agentic AI Layer
The agent can still use live tools where necessary, but the data platform provides a governed foundation.
21. Medallion Is a Logical Architecture
This distinction is critical.
Bronze, Silver, and Gold are logical layers.
They do not necessarily mean:
Three physical clusters
Three storage accounts
Three databases
Three workspaces
You can implement them using different catalogs, schemas, tables, or data products depending on governance and organizational needs.
Databricks explicitly describes medallion as a data design pattern, not a mandatory implementation requirement.
22. Catalog Design
A common question is:
Should I create separate catalogs for Bronze, Silver, and Gold?
There is no universal answer.
Possible approaches include:
Layer-oriented
catalog
├── bronze
├── silver
└── gold
Environment-oriented
dev
├── bronze
├── silver
└── gold
prod
├── bronze
├── silver
└── gold
Domain-oriented
sales
├── bronze
├── silver
└── gold
finance
├── bronze
├── silver
└── gold
Hybrid
prod_sales
├── bronze
├── silver
└── gold
prod_finance
├── bronze
├── silver
└── gold
Databricks' current Unity Catalog guidance supports different organizational models, including environment-based and domain-based approaches, and explicitly notes that naming conventions should be defined by the architecture team.
23. Domain-Oriented Medallion Architecture
For large enterprises, a single centralized Bronze/Silver/Gold hierarchy can become difficult to govern.
Consider:
Enterprise Data Platform
│
┌───────────┼───────────┐
▼ ▼ ▼
Sales Finance HR
│ │ │
B/S/G B/S/G B/S/G
Each domain owns its pipelines and data products.
This moves the architecture closer to Data Mesh principles.
24. Hub-and-Spoke Medallion Architecture
Databricks also documents a hub-and-spoke medallion architecture for enterprise deployments.
Conceptually:
DATA HUB
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Bronze Silver Gold
│
│
Shared Data Products
│
┌─────┼─────────┐
▼ ▼ ▼
Sales Finance Engineering
│ │ │
B/S/G B/S/G B/S/G
The central hub provides organization-wide data assets.
Domains can then combine shared data with their own domain-specific data.
This is particularly useful when:
- Multiple business units exist
- Domains require ownership
- Some data assets are enterprise-wide
- Governance needs to remain centralized
- Data products need controlled sharing
25. Data Products Are More Important Than Tables
A mature architecture should stop thinking only in terms of tables.
Think:
Data Product
A data product should have:
Owner
Definition
Schema
Quality expectations
SLA
Freshness
Lineage
Access policy
Documentation
Consumers
Versioning strategy
For example:
Customer 360 Data Product
Owner:
Customer Domain
Inputs:
CRM
Billing
Support
Web Events
Quality:
99.9% valid customer IDs
Freshness:
< 30 minutes
Consumers:
BI
ML
AI Agents
This is much more valuable than simply saying:
silver.customer
26. Batch vs Streaming
Medallion Architecture supports both.
Batch
Source
↓
Bronze
↓
Silver
↓
Gold
Streaming
Event Source
↓
Streaming Bronze
↓
Streaming Silver
↓
Streaming Gold
↓
Real-Time Consumers
For example:
Kafka
↓
Bronze
↓
Silver
↓
Gold
↓
Real-time Dashboard
The architectural principle remains the same:
Progressively improve data quality and usability.
The processing mode changes.
27. Lakeflow and Medallion Pipelines
Databricks' current pipeline ecosystem uses Lakeflow Declarative Pipelines for building data pipelines, with support for streaming tables, materialized views, data quality expectations, monitoring, and Unity Catalog integration.
A conceptual pipeline could look like:
@bronze
Raw Ingestion
↓
@silver
Validated Dataset
↓
@gold
Business Dataset
The important point isn't the syntax.
It is the separation of responsibilities.
28. Governance Across the Medallion Layers
Governance cannot be an afterthought.
A production architecture needs to answer:
Who owns this data?
Who can read it?
Who can modify it?
Where did it come from?
Who consumed it?
What transformations occurred?
What sensitive data does it contain?
How long should it be retained?
Unity Catalog provides a centralized governance layer for data and AI assets, including permissions, discovery, and lineage capabilities.
29. Governance Should Increase With Data Accessibility
A useful architectural principle is:
Bronze
│
│ Restricted
▼
Silver
│
│ Controlled
▼
Gold
│
│ Broad business access
▼
Consumers
This doesn't mean Gold should automatically be public.
Instead:
Access should be aligned with data sensitivity, business purpose, and ownership.
For example, PII may exist in Bronze and Silver but should not automatically propagate into every Gold dataset.
30. PII and Sensitive Data
Consider:
email
phone
address
SSN
medical_record_number
The architecture should explicitly decide:
- Where sensitive data enters
- Who can access it
- Whether it should be masked
- Whether it should be tokenized
- Which downstream datasets require it
- Whether Gold should contain it at all
For example:
Bronze
Raw PII
↓
Silver
Tokenized / governed PII
↓
Gold
Business-safe identifiers
This reduces unnecessary exposure.
31. Lineage
A business user asks:
"Where did this revenue KPI come from?"
A mature platform should answer:
Revenue KPI
↓
Gold Revenue Table
↓
Silver Transactions
↓
Bronze Transaction Events
↓
ERP
This is one of the reasons governance and lineage are fundamental architectural concerns.
32. Performance Engineering
Medallion Architecture alone does not guarantee performance.
Architects still need to think about:
- Data layout
- File sizes
- Clustering
- Partitioning strategy
- Query patterns
- Incremental processing
- Data skipping
- Compute sizing
- Workload isolation
Current Databricks Delta Lake guidance recommends features such as liquid clustering and predictive optimization for applicable managed-table workloads.
33. Don't Automatically Partition Everything
A common data-platform anti-pattern is:
"We should partition every table."
Not necessarily.
Partitioning should be driven by:
- Query patterns
- Data volume
- Cardinality
- Data distribution
- Maintenance cost
Modern Delta Lake capabilities can reduce the need for traditional partition-heavy designs.
The architecture should optimize for the workload rather than follow a blanket rule.
34. Data Freshness Is an Architectural Requirement
Every data product should have a freshness expectation.
Examples:
Executive Dashboard
→ Daily
Sales Dashboard
→ Hourly
Fraud Detection
→ Seconds / Minutes
Customer 360
→ 15 minutes
AI Knowledge Base
→ Eventual / Scheduled
ML Features
→ Depends on model
The Medallion architecture should therefore be designed around SLAs/SLOs, not merely data movement.
35. Failure Handling
Imagine:
Bronze succeeds
Silver fails
Gold never runs
The architecture should make this state observable.
You need:
- Pipeline monitoring
- Retry strategies
- Dead-letter/quarantine handling
- Alerting
- Data quality metrics
- Processing checkpoints
- Idempotent transformations
A production data architecture must answer:
What happens when the pipeline fails halfway through?
36. Idempotency
Suppose a pipeline processes:
1,000,000 records
and fails after:
750,000
When restarted, you don't want:
750,000 duplicates
The pipeline should be designed so repeated processing produces the correct result.
This is where concepts such as:
- MERGE
- Deduplication
- Checkpoints
- Event IDs
- Batch IDs
- Watermarks
become important.
37. Late-Arriving Data
Consider an event:
Event Time:
08:00
Arrival Time:
08:20
If your Gold aggregation ran at 08:10, the event wasn't available yet.
Your architecture needs a strategy for:
- Late-arriving events
- Watermarks
- Reprocessing
- Backfills
- Correcting aggregates
This is especially important for streaming systems.
38. Backfills
Imagine a business rule changes:
Old definition:
Revenue = completed orders
New definition:
Revenue = completed orders - refunds
If the architecture cannot replay historical data, you may be forced to rebuild the data from operational systems.
A strong Medallion design supports:
Bronze History
↓
Reprocess Silver
↓
Recompute Gold
This is another reason why preserving Bronze matters.
39. Schema Evolution
Source systems change.
Today:
customer
├── id
├── name
└── email
Tomorrow:
customer
├── id
├── name
├── email
└── loyalty_tier
The architecture needs to distinguish:
Expected schema evolution
from:
Unexpected breaking schema change
Bronze can provide a flexible ingestion boundary.
Silver should establish stronger schema expectations.
Gold should expose stable business contracts.
40. Data Contracts
For enterprise architecture, data contracts become increasingly important.
A data contract can define:
Schema
Semantics
Quality
Ownership
Freshness
Compatibility
SLA
For example:
dataset: customer
owner: customer-domain
freshness: 15m
quality:
customer_id: not_null
email: valid_email
compatibility:
mode: backward_compatible
This turns data pipelines from informal integrations into governed interfaces.
41. Medallion Architecture and Data Mesh
These architectures solve different problems.
Medallion Architecture answers:
How should data progressively become more refined?
Data Mesh answers:
How should data ownership and responsibility be organized across domains?
They can work together.
Data Mesh
Domain Ownership
│
┌───────┼────────┐
▼ ▼ ▼
Sales Finance HR
│ │ │
B/S/G B/S/G B/S/G
The Medallion pattern operates inside each domain.
42. Medallion Architecture vs Data Warehouse
They aren't necessarily competing architectures.
A lakehouse can use:
Bronze
↓
Silver
↓
Gold
↓
Data Marts
Silver may contain warehouse-style relational models.
Gold may expose specialized marts.
Databricks documentation explicitly describes scenarios where warehouse-style modeling can occur in Silver and specialized data marts can be created in Gold.
43. Medallion Architecture vs Lambda Architecture
Lambda Architecture traditionally separates:
Batch Layer
+
Speed Layer
+
Serving Layer
Medallion instead focuses on:
Data Quality / Refinement
Bronze
Silver
Gold
They address different dimensions.
A modern platform can support streaming and batch processing through the same logical Medallion layers.
44. A Reference Enterprise Architecture
Putting everything together:
SOURCE SYSTEMS
│
┌───────────────────┼───────────────────┐
│ │ │
Databases APIs Events
│ │ │
└───────────────────┼───────────────────┘
▼
┌──────────────────┐
│ BRONZE │
│ │
│ Raw / Replayable │
│ Source Fidelity │
└────────┬─────────┘
│
Quality + Standardize
│
▼
┌──────────────────┐
│ SILVER │
│ │
│ Validated │
│ Canonical │
│ Enriched │
└────────┬─────────┘
│
Business Modeling
│
▼
┌──────────────────┐
│ GOLD │
│ │
│ Data Products │
│ Metrics │
│ Aggregates │
└────────┬─────────┘
│
┌──────────────────┼───────────────────┐
▼ ▼ ▼
BI ML AI
│ │ │
Dashboards Features/Models RAG/Agents
Across all layers:
┌─────────────────────────────┐
│ UNITY CATALOG │
│ │
│ Governance │
│ Access Control │
│ Discovery │
│ Lineage │
│ Data Sharing │
└─────────────────────────────┘
45. A More Mature Architecture: Hub + Domains
For a large enterprise:
ENTERPRISE DATA HUB
│
┌─────────────┼─────────────┐
▼ ▼ ▼
Shared Shared Shared
Bronze Silver Gold
│
┌───────────┼──────────────┐
▼ ▼ ▼
SALES FINANCE CUSTOMER
│ │ │
B/S/G B/S/G B/S/G
│ │ │
└───────────┼──────────────┘
▼
Enterprise AI
│
┌─────────┼─────────┐
▼ ▼ ▼
RAG ML Models Agents
Databricks' hub-and-spoke guidance explicitly describes this model, where shared organization-wide data is managed centrally while domains can maintain their own domain-specific raw and curated data.
46. Common Architectural Mistakes
Mistake 1: Treating Bronze as a temporary staging area
Bronze should generally provide durable source fidelity and replayability.
Mistake 2: Putting business logic everywhere
If business logic is duplicated across:
BI
ML
Applications
Gold tables
you will eventually get inconsistent definitions.
Centralize reusable semantics where appropriate.
Mistake 3: Letting consumers directly depend on Bronze
Bronze is not designed to be a stable business interface.
Mistake 4: Creating Gold tables for every dashboard
This can create:
Dashboard A → Gold A
Dashboard B → Gold B
Dashboard C → Gold C
with duplicated transformations.
Instead, identify reusable data products and semantic models.
Mistake 5: Treating Silver as just a cleanup layer
Silver can become the organization's canonical, reusable data foundation.
Mistake 6: Ignoring data quality
A beautifully designed architecture with bad data is still a bad architecture.
Mistake 7: Ignoring ownership
Every important dataset should have an accountable owner.
Mistake 8: Creating excessive catalogs
Too many catalogs can create unnecessary administrative complexity. Databricks recommends keeping catalog structures manageable and choosing a consistent organizational model.
Mistake 9: Assuming every dataset needs Bronze → Silver → Gold
Medallion is a pattern, not a law.
Some datasets may legitimately skip or combine layers depending on their lifecycle and use case.
47. When Should You NOT Use Medallion?
This is an important architectural question.
Don't introduce three layers simply because:
"Databricks recommends Medallion."
For a small application with:
10 GB data
2 consumers
One pipeline
No regulatory requirements
No ML
No complex transformations
a full enterprise-style architecture may be unnecessary.
Architecture should follow:
Complexity
Volume
Velocity
Governance
Number of consumers
Data lifecycle
Business criticality
not fashion.
Databricks itself describes Medallion as a recommended best practice rather than a mandatory requirement.
48. The Architect's Decision Framework
When designing a Medallion architecture, ask:
Data
- What are the sources?
- Batch or streaming?
- CDC or snapshots?
- Structured or unstructured?
Bronze
- Can the source data be replayed?
- Are ingestion metadata captured?
- How is schema drift handled?
- What is the retention policy?
Silver
- What constitutes valid data?
- What is the canonical model?
- Where is deduplication performed?
- How are late events handled?
- How is CDC resolved?
Gold
- Who consumes the data?
- What business metrics are required?
- What aggregates are needed?
- Which datasets are reusable data products?
Governance
- Who owns the dataset?
- Who can access it?
- What PII exists?
- What lineage is required?
Operations
- What is the freshness SLA?
- What happens when the pipeline fails?
- How are backfills performed?
- How are quality failures monitored?
AI/ML
- Is this data used for training?
- Is point-in-time correctness required?
- Does it feed RAG?
- Does an agent need real-time access?
- Should the agent call a data product or an operational API?
49. Medallion Architecture for AI-Native Enterprises
As organizations move toward GenAI and Agentic AI, the traditional architecture:
Data → BI
is becoming:
Data
│
├── BI
│
├── ML
│
├── RAG
│
└── Agentic AI
This makes the quality and governance of the underlying data even more important.
An AI agent can reason extremely well.
But if the underlying data is:
- stale
- duplicated
- inconsistent
- poorly governed
- incorrectly transformed
the agent will still produce unreliable outcomes.
This leads to an important architectural principle:
AI quality is constrained by data quality and data accessibility.
50. The Future: From Data Layers to Data Products
The evolution can be thought of as:
Data Lake
↓
Lakehouse
↓
Medallion Architecture
↓
Governed Data Products
↓
ML / GenAI / Agentic AI
The ultimate goal isn't Bronze, Silver, and Gold themselves.
The goal is:
Reliable, discoverable, governed, reusable data products that can serve multiple workloads.
Medallion Architecture is one of the mechanisms that helps organizations get there.
51. Final Architecture Checklist
Before calling a Databricks Medallion implementation production-ready, ask:
□ Raw data is preserved
□ Replay/reprocessing is possible
□ CDC strategy is defined
□ Schema evolution is controlled
□ Data quality rules are explicit
□ Invalid records are handled
□ Canonical models are defined
□ Business definitions are standardized
□ Gold datasets have clear consumers
□ Data products have owners
□ Governance is implemented
□ Lineage is available
□ PII handling is defined
□ Freshness SLAs exist
□ Pipeline failures are observable
□ Backfill strategy exists
□ Streaming strategy is defined
□ Compute/workload isolation is considered
□ ML consumption is supported
□ AI/RAG consumption is supported
□ Agent/tool access is governed
Conclusion
The simplest way to explain Medallion Architecture is:
BRONZE
Preserve the data
↓
SILVER
Trust the data
↓
GOLD
Turn data into business value
But for an architect, the real story is much deeper.
Medallion Architecture provides a framework for establishing:
Source fidelity → Data quality → Canonical semantics → Business context → Governed data products
And those data products can ultimately power:
BI
│
├── Analytics
│
├── Machine Learning
│
├── Generative AI
│
└── Agentic AI
The biggest architectural lesson is therefore not:
"Always use Bronze, Silver, and Gold."
It is:
"Design explicit boundaries for data quality, ownership, governance, replayability, and consumption."
Bronze protects your source fidelity.
Silver establishes trustworthy and reusable data.
Gold turns that data into business-oriented products.
Unity Catalog provides governance and discoverability across the platform.
And together, these patterns can form a strong foundation for modern Lakehouse + ML + GenAI + Agentic AI architectures.
The best architecture, however, is not the one with the most layers.
It's the one that creates the right boundaries for the complexity your organization actually has.
💬 What would you choose?
If you were designing a Databricks platform for a large enterprise, would you choose:
Centralized Medallion → Domain-oriented Medallion → Hub-and-Spoke → Data Mesh + Medallion?
The answer depends heavily on organization structure, governance requirements, data ownership, workload patterns, and AI/ML strategy.
I'd love to hear how other architects approach this.
📚 References
The architecture and current Databricks terminology discussed in this article are based primarily on Databricks' official architecture documentation, including its Medallion Architecture, Delta Lake, Unity Catalog, Lakeflow pipeline, and governance guidance.
Top comments (0)