Data Quality Monitoring for Snowflake and Databricks: A Practical Guide

Last Tuesday a source team renamed order_status to status in Snowflake. No announcement. Your dbt models kept running, but every query referencing order_status silently returned NULLs. The revenue dashboard showed a 40% drop. It took four hours to trace it back to a one-word column rename.

Schema changes like this are the #1 cause of silent data pipeline failures, and neither Snowflake nor Databricks will warn you when they happen. Data quality monitoring catches these problems automatically. Here's how to set it up for the two most common cloud data platforms.

Four types of data quality monitoring

Data quality monitoring breaks down into four categories:

Schema change detection. A column gets added, removed, renamed, or changes type. Your query doesn't error. It just returns NULLs or wrong values.

Freshness monitoring. A daily table hasn't updated in 36 hours. Something upstream is broken and nobody knows yet.

Anomaly detection. Row count drops 80% overnight. Null rate in a critical column spikes from 2% to 40%. Statistical anomalies surface data quality issues before dashboards break.

Correctness checks. Is order_total ever negative? Does every customer_id in the fact table exist in the dimension table? Business-logic validations specific to your data.

Data quality monitoring in Snowflake

Schema tracking

Snowflake's INFORMATION_SCHEMA tracks table and column metadata:

SELECT table_name, column_name, data_type, ordinal_position
FROM information_schema.columns
WHERE table_schema = 'PUBLIC'
ORDER BY table_name, ordinal_position;

The limitation: this shows current state, not change history. To detect changes, you need to snapshot metadata on a schedule and diff it.

Freshness

INFORMATION_SCHEMA.TABLES.LAST_ALTERED gives you the last DDL or DML timestamp. For row-level freshness:

SELECT MAX(updated_at) as last_data_timestamp
FROM my_schema.my_table;

Where Snowflake's built-in tools fall short

• Time Travel provides 90 days of history but querying it at scale for monitoring is expensive.
• Streams and Tasks detect changes but require setup per table with no unified view.
• Dynamic Tables handle some freshness concerns but don't cover schema changes or anomalies.

Data quality monitoring in Databricks

Schema tracking

Delta Lake enforces schemas by default. DESCRIBE HISTORY shows schema evolution over time:

DESCRIBE HISTORY my_catalog.my_schema.my_table;

This is more useful than Snowflake's current-state-only metadata because you can see what changed and when.

Freshness

Delta table history includes commit timestamps:

DESCRIBE HISTORY my_table LIMIT 1;

Where Databricks' built-in tools fall short

• Unity Catalog focuses on access control, not data quality monitoring.
• Lakehouse Monitoring (preview) provides profiling but is limited to Databricks-native tables.
• Delta Live Tables expectations handle correctness within DLT pipelines but don't cover freshness or schema drift across your broader data estate.

What built-in tools miss

Both platforms provide the primitives: metadata queries, change history, schema enforcement. Turning those primitives into monitoring requires:

• Scheduling checks across hundreds of tables every 15-60 minutes
• Diffing current state against historical snapshots to detect changes
• Per-table thresholds because a real-time events table and a weekly rollup need different SLAs
• Alert routing to the team that owns each pipeline, not a shared channel everyone ignores
• Cross-platform visibility because most teams run Snowflake and Databricks alongside BigQuery, Redshift, or PostgreSQL. Monitoring each platform separately means blind spots at the boundaries.

Building this yourself is the common starting point. A scheduled SQL job checks freshness, a Python script diffs schemas, a cron sends Slack messages. It works at 20 tables. At 200 tables across three databases, it becomes its own engineering project. (The 2026 State of Data Engineering Survey found that this kind of reactive maintenance consumes 60% of data engineering time.)

This is where dedicated monitoring tools earn their keep. Instead of maintaining a patchwork of scripts, you connect your warehouses once and get schema change detection, freshness monitoring, and anomaly alerts across everything. AnomalyArmor does exactly this for Snowflake, Databricks, BigQuery, Redshift, and PostgreSQL. Email [e-mail] for a trial code.

Getting started

If you're evaluating data quality monitoring for Snowflake or Databricks:

1. Inventory your critical tables. Which 10-20 tables would cause the most pain if they broke silently?
2. Set up schema change detection first. Highest value, lowest effort. Schema changes cause silent failures that are expensive to debug after the fact.
3. Add freshness monitoring for daily-batch tables. The most common source of "the dashboard is showing yesterday's data" incidents.
4. Layer in anomaly detection once you have baseline data. Anomaly detection needs history to establish normal patterns. Start collecting now.

---

---

Snowflake and Databricks Data Quality FAQ

How is data quality monitoring different in Snowflake vs Databricks?

Snowflake exposes metadata through INFORMATION_SCHEMA and ACCOUNT_USAGE, which makes schema tracking and freshness checks straightforward via SQL. Databricks uses Unity Catalog plus Delta Live Tables expectations, which puts data quality checks inside the pipeline itself. Both platforms support custom SQL monitors but have different strengths: Snowflake is better for cross-database monitoring, Databricks is better for streaming and pipeline-level quality gates.

What is schema drift in Snowflake?

Schema drift in Snowflake refers to unexpected changes in table structure: columns added, dropped, renamed, or having their types changed. Snowflake's INFORMATION_SCHEMA.COLUMNS view tracks all column changes via the LAST_ALTERED timestamp, which makes schema drift detection possible with a scheduled query that compares current state against a snapshot.

How do Delta Live Tables expectations work?

Delta Live Tables (DLT) expectations are SQL predicates attached to tables in a pipeline. You write a check like @dlt.expect("valid_id", "id IS NOT NULL") and DLT enforces it on every row. Rows that fail can be dropped, logged, or cause the pipeline to fail. It's data quality enforcement built into the pipeline, rather than monitoring after the fact.

Can I use the same data quality tool for Snowflake and Databricks?

Yes. Tools like AnomalyArmor, Monte Carlo, Metaplane, and Soda work across Snowflake, Databricks, BigQuery, and PostgreSQL. They abstract the warehouse-specific metadata queries behind a unified interface. This is important if you use both platforms, because maintaining separate monitoring systems for each creates operational overhead.

What are Snowflake Data Metric Functions?

Snowflake Data Metric Functions (DMFs) are built-in functions that measure specific data quality attributes: null count, distinct count, duplicate count, freshness, and custom metrics you define. They run as scheduled jobs on a table and store results in a metadata table. DMFs are native to Snowflake but limited to the platform and lack alert routing or cross-table analysis.

How do I monitor data quality across multiple warehouses?

Use a tool that connects to all your warehouses and provides a unified view. Running separate monitoring per warehouse creates blind spots at the boundaries: when data moves from Snowflake to Databricks to a downstream warehouse, you need end-to-end visibility. Cross-warehouse tools handle schema drift tracking, freshness monitoring, and anomaly detection in one place.

What's the cost of data quality monitoring in Snowflake?

Monitoring queries consume Snowflake credits. Schema metadata queries are cheap (milliseconds). Statistical queries on large tables can be expensive if run frequently. Most monitoring tools optimize by sampling, caching, and running queries on off-peak warehouses. Budget 1-5% of your Snowflake spend for monitoring queries if you monitor 50+ critical tables.

Should I use dbt tests or a data observability tool?

dbt tests are good for deterministic validation: "this column should never be null", "these two tables should have matching row counts". Data observability tools are good for statistical anomaly detection, schema drift tracking, freshness monitoring, and cross-table analysis. Most teams use both: dbt tests inside the transformation layer, and observability tools for production monitoring across all data sources.

How do I detect when a Databricks table stops updating?

Query information_schema.tables for the last_altered timestamp, or check the Delta transaction log for the most recent commit. Compare against a freshness SLA. You can also watch for a gap between the expected job schedule and the actual last update time. Dedicated monitoring tools automate this by tracking per-table freshness patterns.

What's the minimum setup time for data quality monitoring?

For a single table with basic checks (row count, null rate, schema), you can set up monitoring in under 10 minutes with most tools. For comprehensive coverage across 50+ tables with alert routing, SLAs, and seasonality detection, budget 1-2 days of configuration. Tools with AI-powered setup (like AnomalyArmor's agent) can configure monitoring from natural language in minutes.

---

AnomalyArmor monitors Snowflake, Databricks, PostgreSQL, BigQuery, and more from a single interface. See how the AI agent configures monitoring in seconds.