DEV Community: Blaine Elliott

Data Anomaly Detection: The Complete Guide for Data Engineers

Blaine Elliott — Sat, 11 Apr 2026 22:48:30 +0000

Data anomaly detection is the process of identifying data points, patterns, or values that deviate from expected behavior. It catches schema changes, stale tables, row count spikes, and statistical outliers before they break dashboards or corrupt downstream analytics. Modern data anomaly detection combines statistical methods like z-scores and Welford's algorithm with machine learning models that learn seasonal patterns from historical data.

This guide explains the four types of data anomalies, the algorithms used to detect each one, and how to implement detection in Snowflake, Databricks, and PostgreSQL.

What is data anomaly detection?

Data anomaly detection is the automated identification of unexpected values, patterns, or changes in a dataset. In data engineering, it monitors production tables for problems like:

A column gets renamed, dropped, or changes type (schema drift)
A daily-updated table hasn't received new rows in 36 hours (freshness failure)
Row counts drop by 80% overnight (volume anomaly)
Null rate in a critical column spikes from 2% to 40% (quality anomaly)
A customer ID in a fact table references a non-existent record (referential anomaly)

The goal is to catch these problems before they reach dashboards, ML models, or customer-facing applications.

The four types of data anomalies

1. Schema anomalies

Schema anomalies occur when the structure of a table changes unexpectedly. Common examples:

Column added: A new column appears upstream, which can break SELECT * queries
Column dropped: A column disappears, breaking any query that references it
Column renamed: The column exists under a different name, causing silent NULL returns
Type changed: A VARCHAR becomes an INTEGER, causing cast failures

Schema anomalies are the most common cause of silent data failures because queries often continue to run without error, returning wrong results.

2. Freshness anomalies

Freshness anomalies happen when a table stops updating on its expected schedule. A table that normally updates every hour but hasn't received new rows in 6 hours has a freshness anomaly. These are caused by:

Upstream pipeline failures
Source system outages
Broken scheduled jobs
Permission changes

Freshness is typically measured as "time since last insert" or "max(timestamp_column)".

3. Volume anomalies

Volume anomalies are unexpected changes in row counts. A daily sales table that normally receives 10,000-12,000 rows suddenly receiving 500 rows (or 100,000) is a volume anomaly. Causes include:

Upstream filter changes
Duplicate data ingestion
Failed partial loads
Fraud or bot activity

4. Value anomalies

Value anomalies are statistical outliers in column values. Examples:

A revenue column where 5% of rows are negative when they should always be positive
A foreign key column where null rates spike from 2% to 40%
A timestamp column with future dates

Value anomalies are detected using statistical methods applied to specific columns.

How data anomaly detection works

Anomaly detection uses three main approaches: static thresholds, statistical methods, and machine learning.

Static thresholds

The simplest approach. You define the expected range manually:

SELECT 'anomaly' AS status
FROM orders
WHERE COUNT(*) < 1000 OR COUNT(*) > 50000;

Static thresholds work for stable metrics but fail for anything with seasonality (weekend traffic drops, end-of-month spikes).

Statistical methods

Statistical anomaly detection uses historical data to compute expected ranges automatically. The most common approach is the z-score:

z = (current_value - historical_mean) / historical_stddev

If the absolute z-score exceeds a threshold (typically 2 or 3), the value is flagged as anomalous. A z-score of 2 catches values more than 2 standard deviations from the mean, which is roughly the top or bottom 2.5% of a normal distribution.

Welford's algorithm is the most efficient way to compute running mean and standard deviation for anomaly detection. It maintains three numbers (count, mean, and sum of squared deviations) and updates them incrementally with each new data point, requiring constant memory:

def update_stats(count, mean, m2, value):
    count += 1
    delta = value - mean
    mean += delta / count
    delta2 = value - mean
    m2 += delta * delta2
    return count, mean, m2

def get_variance(count, m2):
    return m2 / (count - 1) if count > 1 else 0

This is the foundation of most production anomaly detection systems because it scales to high-volume event streams without storing historical data.

Machine learning methods

For data with complex seasonality (weekly patterns, business hours, holiday effects), machine learning models outperform simple statistics. The most common approach is Prophet (Facebook's time-series forecasting library), which decomposes a series into trend, weekly seasonality, and yearly seasonality, then flags values outside the prediction interval.

Prophet requires at least 14 data points to detect weekly patterns and 365 points to detect yearly patterns. For tables with less history, fall back to z-scores.

How to detect data anomalies in Snowflake

Snowflake provides metadata views that make anomaly detection straightforward.

Schema anomalies: Track column changes via INFORMATION_SCHEMA.COLUMNS:

SELECT table_name, column_name, data_type, last_altered
FROM information_schema.columns
WHERE table_schema = 'PRODUCTION'
  AND last_altered > DATEADD(hour, -24, CURRENT_TIMESTAMP());

Freshness anomalies: Check ACCOUNT_USAGE.TABLES for last DML operation:

SELECT table_name,
       last_altered,
       DATEDIFF(hour, last_altered, CURRENT_TIMESTAMP()) AS hours_stale
FROM snowflake.account_usage.tables
WHERE table_schema = 'PRODUCTION'
  AND DATEDIFF(hour, last_altered, CURRENT_TIMESTAMP()) > 24;

Volume anomalies: Compare today's row count against a rolling 30-day average:

WITH daily_counts AS (
  SELECT DATE(created_at) AS day, COUNT(*) AS row_count
  FROM orders
  WHERE created_at >= DATEADD(day, -30, CURRENT_DATE())
  GROUP BY DATE(created_at)
),
stats AS (
  SELECT AVG(row_count) AS mean, STDDEV(row_count) AS stddev
  FROM daily_counts
  WHERE day < CURRENT_DATE()
)
SELECT row_count,
       (row_count - mean) / stddev AS z_score
FROM daily_counts, stats
WHERE day = CURRENT_DATE()
  AND ABS((row_count - mean) / stddev) > 2;

How to detect data anomalies in Databricks

Databricks offers Delta Live Tables expectations for inline anomaly detection:

import dlt
from pyspark.sql.functions import col

@dlt.table
@dlt.expect_or_drop("valid_order_total", "order_total > 0")
@dlt.expect_or_fail("recent_data", "created_at > current_date() - interval 2 days")
def clean_orders():
    return spark.read.table("raw_orders")

For volume and statistical anomalies, use Unity Catalog's lineage tracking combined with scheduled queries:

SELECT table_name,
       COUNT(*) AS row_count,
       MAX(ingestion_time) AS last_update
FROM production.orders
GROUP BY table_name;

How to detect data anomalies in PostgreSQL

PostgreSQL doesn't have built-in anomaly detection, but you can implement it with pg_stat_user_tables and custom queries:

SELECT relname AS table_name,
       n_live_tup AS row_count,
       last_autoanalyze
FROM pg_stat_user_tables
WHERE schemaname = 'public'
  AND last_autoanalyze < NOW() - INTERVAL '24 hours';

For value anomalies, use window functions to compute rolling statistics:

WITH rolling_stats AS (
  SELECT order_id,
         amount,
         AVG(amount) OVER (ORDER BY created_at ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING) AS rolling_mean,
         STDDEV(amount) OVER (ORDER BY created_at ROWS BETWEEN 30 PRECEDING AND 1 PRECEDING) AS rolling_stddev
  FROM orders
)
SELECT order_id, amount, rolling_mean, rolling_stddev,
       (amount - rolling_mean) / NULLIF(rolling_stddev, 0) AS z_score
FROM rolling_stats
WHERE ABS((amount - rolling_mean) / NULLIF(rolling_stddev, 0)) > 3;

Build vs buy: data anomaly detection tools

Building anomaly detection in-house gives you control but requires engineering time to maintain. Most data teams outgrow custom solutions because:

Alert fatigue: Static thresholds fire too often and get ignored
Seasonality blindness: Simple statistics miss weekly and yearly patterns
Cross-platform monitoring: Different code for Snowflake, Databricks, and Postgres
Incident triage: No unified view of which alerts matter most

AnomalyArmor is a data observability platform that uses AI to configure anomaly detection automatically. You connect your data warehouse, describe what you want to monitor in plain English, and the AI agent sets up schema drift alerts, freshness schedules, and statistical anomaly detection across all your tables. It works on Snowflake, Databricks, PostgreSQL, and BigQuery.

Data anomaly detection FAQ

What is the difference between anomaly detection and data validation?

Data validation checks if data matches explicit rules (e.g., "order_id is not null"). Anomaly detection uses statistical methods to identify values that deviate from historical patterns. Validation catches known problems. Anomaly detection catches unknown ones.

What is the best algorithm for data anomaly detection?

For most production use cases, z-scores computed with Welford's algorithm work well. For data with strong weekly or yearly seasonality, Prophet or similar time-series models are better. For high-dimensional data, isolation forests outperform statistical methods.

How do I detect schema drift automatically?

Query your database's INFORMATION_SCHEMA or metadata views on a schedule, store the previous state, and diff the current state against the stored version. When columns change, type definitions change, or tables are added or removed, fire an alert. AnomalyArmor does this automatically for Snowflake, Databricks, and PostgreSQL.

What is a z-score and how is it used in anomaly detection?

A z-score measures how many standard deviations a value is from the historical mean. A z-score of 2 means the value is 2 standard deviations above the mean, which occurs in roughly 2.5% of a normal distribution. Most anomaly detection systems use z-scores between 2 and 3 as thresholds.

How much historical data do I need for anomaly detection?

Statistical methods like z-scores need at least 7-10 data points to produce meaningful baselines. Machine learning methods like Prophet need at least 14 points for weekly seasonality and 365 points for yearly seasonality. During the learning phase, most systems don't fire alerts.

What is the difference between data observability and anomaly detection?

Anomaly detection is one component of data observability. Data observability also includes lineage tracking, impact analysis, schema change detection, and root cause analysis. Anomaly detection tells you something is wrong. Observability tells you what, where, and why.

Can AI improve data anomaly detection?

Yes. AI improves anomaly detection in three ways. First, AI agents can configure monitoring rules from natural language instead of YAML or GUI forms. Second, LLMs can analyze alert patterns to reduce false positives. Third, AI can correlate anomalies across tables to identify root causes faster than manual investigation.

How do I avoid alert fatigue in anomaly detection?

Use adaptive thresholds that learn from historical patterns instead of static rules. Set sensitivity per table based on how critical it is. Group related alerts so a single upstream failure generates one notification instead of ten. Suppress alerts during known maintenance windows.

What data platforms support anomaly detection natively?

Snowflake has data metric functions and ACCOUNT_USAGE views. Databricks has Delta Live Tables expectations and Unity Catalog lineage. BigQuery has table metadata and scheduled queries. PostgreSQL has pg_stat_user_tables. None of these are full anomaly detection systems, but they provide the raw metrics needed to build one.

How real-time should anomaly detection be?

It depends on the use case. Schema drift and freshness checks should run every 5-15 minutes. Row count and statistical anomalies should run hourly for most tables and daily for slower-changing ones. Real-time streaming anomaly detection (sub-second) is rarely needed for data warehouses but is critical for fraud detection and security monitoring.

Summary

Data anomaly detection catches schema changes, freshness failures, volume spikes, and statistical outliers before they break downstream analytics. The four main types of anomalies require different detection approaches: schema changes need metadata diffs, freshness needs time-since-update checks, volume needs historical baselines, and value anomalies need statistical methods like z-scores or machine learning models like Prophet.

Modern data observability platforms combine all four detection methods with AI-powered configuration to make anomaly detection practical at scale. Whether you build in-house or buy a tool, the fundamental algorithms are the same: maintain historical baselines, compute expected ranges, and flag deviations beyond your sensitivity threshold.

Want to see data anomaly detection in action? Watch a 30-second demo of AI configuring schema drift monitoring in real time.

You Don't Need to Write Data Tests

Blaine Elliott — Sat, 11 Apr 2026 22:47:37 +0000

Spend five minutes in any data engineering forum and you'll find the same confession repeated in different words: "We just eyeball row counts and pray." It shows up on Reddit, Hacker News, the dbt Community Forum, Stack Overflow. The phrasing changes but the story doesn't.

Data engineers know they should be testing. They're not skipping tests because they're lazy or because they don't understand the value. They're skipping tests because everything else in their environment conspires against it.

Why data engineers don't test

If you talk to enough practitioners (or read enough forum threads), the same reasons surface over and over:

Nobody gives them time. Organizations reward fast delivery, not reliable delivery. If decision makers don't prioritize testing, it never becomes a standard. The incentive structure actively punishes thoroughness. You get more credit for shipping a pipeline in two days than for spending a week making it bulletproof.

Data changes faster than tests can keep up. This is what separates data testing from software testing. Your code doesn't change overnight. Your data does. A source team renames a column. A third-party API changes its response format. A bulk operation shifts row counts by 40%. Tests written last month don't account for changes that happened last night.

Data quality is invisible until it breaks. The fundamental problem in data engineering is that a bad query still returns results. Results, but not necessarily correct ones. If nobody can see when things are broken, nobody builds the political will to prevent breakage.

Data is inherently hard to test. You can test code. Data is another story. Unit tests verify that your transformation logic works. They don't verify that the data you received is what you expected. These are fundamentally different problems, and the second one causes far more real-world failures.

Code testing vs data testing

This is the distinction the industry has been dancing around for years. Unit tests and data quality checks are different things, and conflating them is why most testing advice falls flat for data teams.

Unit tests verify your code does what you intended. They answer: "Does my transformation produce the right output given known input?"

Data quality checks verify the data you received is what you expected. They answer: "Did 50,000 rows actually arrive? Is the schema the same as yesterday? Are null rates within normal bounds? Did the distribution shift?"

In data engineering, the second category catches far more production failures than the first. Your dbt model can be perfectly correct and still produce garbage if the source data changed underneath it.

Most testing advice aimed at data engineers focuses on the first category. Write unit tests for your transformations. Test your SQL with fixtures. Use dbt tests. This is useful, but it misses the failures that actually page people at 3am.

"Make testing easier" is the wrong frame

The conventional wisdom is: testing is too hard, so let's make it easier. Better frameworks. Better test runners. Better dbt test macros. AI-assisted test generation.

That's genuinely helpful for teams that have the bandwidth to maintain a test suite. But it doesn't address the actual constraint. The problem isn't that testing is too hard. The problem is that testing is another thing to maintain in an environment where there's already not enough time.

Making tests 50% easier to write doesn't help when nobody has time to write them at all. And even if you find time to write them, data changes faster than tests can keep up.

The better frame: don't make testing easier. Make it unnecessary.

Automated data testing: tests you never write

Automated data testing flips the model. Instead of engineers defining what "correct" looks like for every table, the system learns what normal looks like and alerts when something deviates.

This covers the checks that catch the majority of real incidents:

Schema change detection. A column gets renamed, removed, or changes type. This breaks downstream models, joins, and dashboards. You don't need a handwritten test for this. You need a system that tracks schema state and alerts on any change.

Freshness monitoring. A table that updates every hour hasn't been touched in six hours. The pipeline didn't error. It just silently stopped. A system that learns update patterns and flags deviations catches this without any configuration.

Volume anomalies. A table that normally loads 100,000 rows per day suddenly loads 1,000. Or zero. Or 500,000. Anomaly detection against historical baselines catches this without anyone defining thresholds.

Distribution shifts. A column's null rate jumps from 2% to 35%. A numeric field's average drops by half. These are the subtle failures that pass a "did it run?" check but corrupt downstream analytics.

None of these require writing tests. They require connecting to your data warehouse and letting the system build baselines.

What this looks like in practice

You connect your Snowflake, Databricks, BigQuery, PostgreSQL, or Redshift warehouse. The system runs discovery: what tables exist, what schemas they have, when they typically update, what their normal row counts and distributions look like.

From that point, monitoring is automatic. Schema changes trigger alerts. Stale tables trigger alerts. Volume and distribution anomalies trigger alerts. All of this happens without writing a single line of test code.

When something fires, you get context: which table, what changed, when it changed, and which downstream assets are affected. The alert isn't "test failed." The alert is "the orders_fact table hasn't updated in 4 hours, and 12 downstream models depend on it."

This is what AnomalyArmor does. Five-minute setup, no test authoring, no test maintenance. It watches your warehouse and tells you when something looks wrong. The coverage scales with your warehouse, not with your team's bandwidth to write tests. See the quickstart guide to connect your first data source.

This doesn't replace all testing

To be clear: automated data testing doesn't eliminate the need for all handwritten tests. If you have specific business rules (revenue must be positive, email must contain @, every order must have a customer), those still need explicit validation.

But most data teams don't have any testing at all. They're eyeballing row counts and praying. For those teams, automated data testing provides 80% of the coverage with 0% of the authoring effort.

Start with automated monitoring. Add handwritten tests for your most critical business rules. That's the order that matches reality for time-constrained data teams.

The real question

The real question isn't whether every possible scenario has been tested. It's how much uncertainty your organization is willing to tolerate before it starts verifying the numbers it depends on.

For most data teams, the answer has been: a lot of uncertainty. Because the alternative was writing and maintaining tests they didn't have time for.

Automated data testing changes that tradeoff. The cost of coverage drops to near zero. The question stops being "can we afford to test?" and becomes "why aren't we?"

Sources

Joe Reis, 2026 State of Data Engineering Survey (2026). 1,101 respondents. Found data teams spend 34% of time on data quality, 26% on firefighting.
AnomalyArmor, Quickstart Guide. Connect your first data source and set up automated monitoring.
AnomalyArmor, Schema Monitoring Docs. How automated schema change detection works.
AnomalyArmor, Data Quality Monitoring Docs. Volume, distribution, and anomaly monitoring reference.

Automated Data Testing FAQ

What is automated data testing?

Automated data testing is software that continuously validates data without requiring engineers to write explicit test cases. It learns patterns from historical data (volume, schema, distributions, freshness) and alerts when new data deviates from those patterns. It's the opposite of manual test writing like dbt tests or custom SQL assertions.

How is automated data testing different from dbt tests?

dbt tests are deterministic rules you write manually: "this column is unique", "this foreign key exists". Automated data testing learns baselines from historical data and flags statistical deviations. dbt tests catch known problems. Automated testing catches unknown problems. Most production teams use both.

Do I still need to write data tests if I use automated testing?

Yes, for business-critical invariants. Some rules must be enforced explicitly: "revenue must never be negative", "user_id in orders must exist in users". Write these as dbt tests or validation rules. Use automated testing for everything else (statistical anomalies, freshness, schema changes, volume drops).

What can automated data testing detect that manual tests can't?

Automated testing catches things you didn't know to look for: a column's null rate drifting from 2% to 15% over two weeks, row count dropping by 30% on Tuesdays only, a new category appearing in an enum column, a schema change that silently returns NULL for one in a million rows. These are invisible to explicit rules unless you already anticipated them.

Why don't data engineers write more tests?

Three reasons. First, writing tests requires knowing what to test, and data changes faster than test coverage. Second, test maintenance scales linearly with the number of tables, so a team with 500 tables drowns in test code. Third, the ROI of manual tests is unclear until something breaks, so writing them feels like prevention against unknown risks.

How do automated data tests learn what's normal?

They compute baselines from historical data using statistical methods: running mean and standard deviation (often via Welford's algorithm), distribution fingerprints, seasonality models like Prophet, and moving averages. The baselines update incrementally as new data arrives. Most systems require 7-14 days of history before alerts start firing.

What's the false positive rate of automated data testing?

Well-tuned systems run at 5-15% false positive rates using z-scores with sensitivity thresholds of 2-3 standard deviations. Poorly tuned systems can exceed 50%. The key factors are: enough historical data to establish stable baselines, seasonality-aware models for data with weekly or daily patterns, and sensitivity tuning per table based on business criticality.

Can AI replace data engineers writing tests?

AI can configure and maintain monitoring based on patterns it learns from your data. It can't replace business logic validation. A data engineer still needs to specify what matters to the business. But AI removes the grunt work of writing 500 tests for 500 tables, which is where most test-writing effort is wasted.

What tools provide automated data testing?

Leaders in this space include AnomalyArmor, Monte Carlo, Metaplane, Bigeye, and Datafold. Each uses statistical methods to learn baselines and detect anomalies. Open-source options include re_data and Elementary. Traditional tools like Great Expectations require manual test writing but can be combined with profiling to semi-automate.

How much historical data do I need before automated testing works?

Minimum 7 days for basic z-score detection on daily data, 14 days for weekly seasonality detection, and 365 days for yearly seasonality. During the initial learning period, alerts should be suppressed or warnings only. Most tools have a "learning phase" flag that prevents false alerts until the baseline is stable.

Stop writing and maintaining data tests. See how AnomalyArmor's AI agent configures monitoring from a single sentence.

Data Pipeline Monitoring: How to Stop Silent Failures Before They Hit Production

Blaine Elliott — Sat, 11 Apr 2026 22:32:03 +0000

Your Airflow DAG shows all green. Every task completed. No errors in the logs.

But the revenue dashboard is showing yesterday's numbers. A downstream ML model is training on stale features. The finance team is about to close the quarter using incomplete data.

This is the most dangerous type of pipeline failure: the one that doesn't look like a failure at all. And it's far more common than the kind that throws an error.

Data pipeline monitoring exists to catch exactly this. Not job-level "did it run?" checks. Outcome-level "did the data actually arrive, and does it look right?" checks. The difference between those two questions is where most data incidents live.

What is data pipeline monitoring?

Data pipeline monitoring is continuous validation that data is flowing correctly through every stage of your pipeline, from ingestion to transformation to the tables your stakeholders query.

It covers five dimensions:

Freshness: Is data arriving on schedule?
Volume: Are the expected number of rows landing?
Schema: Have columns been added, removed, or changed type?
Distribution: Do the values look normal, or has something shifted?
Lineage: When something breaks, which downstream tables and dashboards are affected?

Most teams start with the first two and add the rest as they scale. But even basic freshness and volume checks catch the majority of incidents that slip past orchestration tools.

The 5 types of pipeline failures (and which ones your tools miss)

1. The successful failure

A DAG runs to completion. Zero errors. But the source API returned an empty response, so the pipeline wrote zero rows. The orchestrator sees a successful run. The table is now empty or stale.

What catches it: Volume monitoring. If a table that normally receives 50,000 rows per load suddenly gets zero, that's an alert.

2. The schema surprise

Someone on the source team renames a column from user_id to userId. Your pipeline doesn't error, it just silently drops the column or fills it with nulls. Downstream joins break. Metrics go wrong. Nobody notices for three days.

What catches it: Schema change detection. Any added, removed, or type-changed column triggers an alert before downstream transformations run.

3. The slow drift

Data volumes gradually decrease by 5% per week. No single day looks alarming. But after a month, you're missing 20% of your records. A filter change upstream, a timezone bug, a partition misconfiguration.

What catches it: Distribution and volume trend monitoring. Anomaly detection that compares today's load against historical patterns, not just a static threshold.

4. The partial load

The pipeline runs, but only processes data from 3 of 5 source partitions. Row counts look lower than normal, but not dramatically. The missing data is from one region, so the aggregate metrics look "close enough" to pass a quick glance.

What catches it: Volume monitoring with granular baselines, comparing expected vs actual row counts at the partition or segment level.

5. The delayed cascade

A source table updates 4 hours late. Downstream transformations ran on schedule and processed stale input. The numbers are technically "fresh" (the downstream table updated on time) but wrong (it used yesterday's source data).

What catches it: Freshness monitoring on source tables, combined with lineage awareness that understands the dependency chain. The downstream table looks fresh, but tracing upstream reveals the root cause.

Why orchestration alerts aren't enough

Airflow, Dagster, Prefect, and similar tools monitor the process: did the job start, run, and finish? They answer "did my code execute?" not "did my data arrive correctly?"

Three specific gaps:

1. Successful jobs that produce wrong output. A job can complete with exit code 0 and write garbage. The orchestrator has no opinion about data content. It ran your code. That's its job.

2. No cross-system visibility. Your pipeline pulls from a Postgres source, transforms in dbt, and lands in Snowflake. The orchestrator sees the dbt run. It doesn't know the Postgres source stopped updating two hours before the dbt run kicked off.

3. No historical baselines. Orchestration tools tell you about this run. They don't tell you whether this run's output looks normal compared to the last 30 runs. A table loading 1,000 rows isn't alarming, unless it normally loads 100,000.

Data pipeline monitoring sits on top of orchestration. It checks what the orchestrator can't: the actual data that landed.

What good data pipeline monitoring looks like

Effective monitoring has four properties:

1. It monitors outcomes, not processes

Check the table, not the job. Did rows arrive? Are the columns intact? Do the values fall within expected ranges? This is the fundamental shift from orchestration monitoring.

2. It adapts to patterns

A static threshold of "alert if fewer than 10,000 rows" breaks when your table legitimately receives 2,000 rows on weekends. Good monitoring learns the pattern and alerts on deviations from it, not from a fixed number.

3. It maps dependencies

When a source table is late, you need to know which downstream tables, dashboards, and reports are affected. Without lineage, you're manually tracing dependencies across systems during an incident, which is the worst time to be doing it.

4. It routes alerts to the right people

A freshness alert on the marketing analytics table should go to the data engineering team that owns that pipeline, not to a shared #data-alerts channel that everyone has muted. Alert routing by ownership turns monitoring from noise into action.

How to set up data pipeline monitoring

Step 1: Identify your critical tables

You don't need to monitor everything on day one. Start with the 10-20 tables that power:

Executive dashboards
Customer-facing data products
Financial reporting
ML model features

These are the tables where a silent failure causes the most damage.

Step 2: Set freshness and volume baselines

For each critical table:

Freshness: How often should this table update? Set the SLA slightly longer than the expected interval. A table that updates hourly gets a 2-hour SLA.
Volume: How many rows does a typical load produce? Set a range based on the last 30 days, accounting for weekday/weekend variation.

Step 3: Enable schema change detection

Schema changes are the most common cause of silent pipeline failures. Any column added, removed, renamed, or type-changed should generate an alert. This catches problems at the source before they propagate downstream.

Step 4: Connect your alert channels

Route alerts to Slack, PagerDuty, or email based on table ownership. The person who gets the alert should be the person who can fix it.

Step 5: Expand gradually

Once your critical tables are monitored, expand to the next tier. Most teams reach full coverage within a few weeks, not months.

The build vs buy decision

You can build basic monitoring with SQL queries and a scheduler. Check INFORMATION_SCHEMA for freshness, run COUNT(*) for volume, compare schemas against a stored baseline.

This works for 5-10 tables. At 50+ tables across multiple databases, you're maintaining:

A custom scheduler running checks every 15-60 minutes
Per-table configurations for thresholds and SLAs
Historical storage for baselines and trend comparison
Alert routing logic by table ownership
A UI for your team to see monitoring status

At that point, the monitoring system is its own engineering project. The question is whether your team's time is better spent maintaining monitoring infrastructure or building data products.

Purpose-built tools like AnomalyArmor handle this out of the box. Connect your warehouse, and freshness, volume, and schema monitoring start automatically. AI-powered analysis explains what changed and why, so you spend less time investigating and more time fixing. Setup takes minutes, not weeks.

Common mistakes to avoid

Setting thresholds too tight. A freshness SLA of 61 minutes on a table that updates hourly will fire every time there's a minor delay. Start generous and tighten over time.

Monitoring everything equally. Not every table is critical. A staging table that only you use doesn't need PagerDuty integration. Prioritize by downstream impact.

Ignoring weekends and holidays. Many pipelines have legitimately different patterns on weekends. Your monitoring needs to account for this or you'll get false alerts every Saturday.

Alert channel sprawl. Sending every alert to a shared Slack channel guarantees they'll be ignored. Route alerts to the specific team that owns the pipeline.

Treating monitoring as a one-time setup. Your pipelines change. New tables get added, old ones get deprecated, schedules shift. Monitoring configuration needs to evolve with your data stack.

FAQ

What's the difference between data pipeline monitoring and data observability?

Data pipeline monitoring focuses on whether data is flowing correctly through your pipelines: freshness, volume, schema. Data observability is the broader discipline that includes monitoring plus lineage, root cause analysis, and historical context. Monitoring is the foundation. Observability is the full picture.

Do I need monitoring if I already use dbt tests?

Yes. dbt tests validate data at transformation time. They check "is this data correct right now?" Monitoring checks "is this data arriving on schedule, in the expected volume, with the expected schema?" They answer different questions. dbt tests catch logic bugs. Monitoring catches infrastructure and upstream failures.

How many tables should I monitor?

Start with your 10-20 most critical tables. Expand from there. Most teams reach full coverage (all production tables) within a few weeks. The goal is 100% coverage of anything that powers a decision, dashboard, or downstream system.

What's the right alert threshold for freshness?

Set it at 1.5-2x your expected update interval. A table that updates every hour should alert at 2 hours. A daily table should alert at 25-26 hours. This avoids false alarms from minor delays while catching real failures.

Can I build my own pipeline monitoring?

You can, and many teams start there. SQL queries checking freshness and row counts are straightforward for a handful of tables. The maintenance burden grows quickly at scale. Most teams that start DIY either invest significant engineering time maintaining it or switch to a purpose-built tool within 6-12 months.

Data Observability vs Data Quality: What's the Difference and Do You Need Both?

Blaine Elliott — Sat, 11 Apr 2026 22:31:15 +0000

Data observability and data quality get used interchangeably, but they solve different problems. Confusing them leads to buying the wrong tool, building the wrong monitors, and missing the issues that actually break things.

Here's the short version: data observability tells you whether your pipelines are working. Data quality tells you whether the data itself is correct. One watches the plumbing. The other checks the water.

Data observability: watching the pipes

Data observability monitors the infrastructure that moves data. It answers questions like:

Did this table update on schedule? (freshness)
Did the number of rows change unexpectedly? (volume)
Did someone add, remove, or rename columns? (schema changes)
Where did this data come from, and what depends on it? (lineage)

These are all things you can measure without knowing anything about what the data means. You don't need business logic. You don't need to know that revenue should always be positive or that email should contain an @ sign. You're just watching patterns and alerting when they break.

Data observability catches problems like:

An Airflow DAG failed silently at 3am and your morning dashboards show stale data
A backend engineer renamed user_id to account_id and broke 12 downstream models
A bulk delete wiped 40% of your rows and nobody noticed for two days
A table that normally updates every hour hasn't been touched in six hours

These are infrastructure failures. The data pipeline broke, and observability tells you where and when.

Data quality: checking the water

Data quality validates the actual content of your data. It answers questions like:

Is email always a valid email address? (validity)
Are there duplicate rows in the orders table? (uniqueness)
Does every order_id in the line items table exist in the orders table? (referential integrity)
Is price always positive? (range/business rules)
Are null rates for critical columns within expected bounds? (completeness)

These checks require domain knowledge. Someone has to decide that price should be positive, that email should match a pattern, that country_code should be in a known list. The tool can automate the checking, but a human has to define what "correct" means.

Data quality catches problems like:

A third-party API started sending prices in cents instead of dollars
A form change allowed empty email addresses into the database
Duplicate records from a retry bug inflated conversion metrics by 15%
A timezone bug shifted all timestamps by 5 hours

These are data content failures. The pipeline worked fine. The data arrived on time, with the right schema, in the right volume. It was just wrong.

Where they overlap

The line between observability and quality isn't always clean. Some examples:

Volume anomalies sit in both camps. A sudden drop in row count could be a pipeline failure (observability) or a business change (quality). The monitoring is the same. The response is different.

Null rate spikes are technically a quality metric, but a sudden increase in nulls for a column that's always been 100% populated usually means something broke upstream. That's an observability signal.

Schema changes are pure observability, but they can cause data quality problems downstream. A column type change from int to varchar might not break the pipeline, but it could produce garbage in your aggregations.

Most modern tools handle both to some degree. The question is emphasis.

When to use which

Start with observability if:

You don't have any monitoring today and want coverage fast
Your biggest pain point is stale dashboards and broken pipelines
You want automated detection without writing rules for every table
You have hundreds of tables and can't manually define quality checks for all of them

Observability tools can start monitoring the day you connect. They learn what "normal" looks like and alert on deviations. No configuration needed for the basics.

Add quality checks when:

You have specific business rules that must always hold (prices > 0, no duplicate orders)
You're dealing with data from external sources you don't control
Regulatory compliance requires you to prove data accuracy
Your data powers ML models where subtle incorrectness compounds

Quality checks are more effort to set up but catch problems that observability misses. A table can be perfectly fresh, with the right schema and normal volume, and still be full of wrong data.

The practical answer: Start with observability for broad coverage, then layer quality checks on your most critical tables. You get 80% of the value from observability with 20% of the setup effort. Quality checks fill the gap for the tables where correctness actually matters.

How the tools stack up

Most tools in this space started on one side and expanded toward the other.

Observability-first tools (AnomalyArmor, Bigeye, Metaplane) give you automated schema, freshness, and volume monitoring out of the box. You connect a database, and within minutes you have baseline coverage across every table. Quality features were added later: custom metrics, validity rules, referential checks.

Governance-first tools (Monte Carlo) started with enterprise data governance, cataloging, and compliance, then expanded into observability and monitoring. They're comprehensive but come with enterprise pricing and longer setup times. If your primary need is pipeline monitoring, you're paying for a lot of surface area you don't use.

Quality-first tools (Great Expectations, Soda, dbt tests) start with explicit validation rules that you write. You define expectations ("this column should never be null," "row count should be between 1000 and 5000") and the tool checks them on a schedule. Observability features like freshness monitoring and lineage are bolted on or require additional setup.

The trend is convergence. Every observability tool now adds quality metrics. Every quality tool now has some form of freshness monitoring. Governance tools are expanding down-market. The difference is which side is mature and which side feels like an afterthought.

What to look for in practice

Skip the category debate and focus on what actually matters:

Time to first alert. How fast can you go from zero monitoring to getting notified when something breaks? If the answer is weeks of configuration, that's a quality-first tool pretending to do observability. If the answer is hours, that's observability-first, which is what you want for starting out.

False positive rate. A tool that alerts on everything is worse than no tool. AI-powered anomaly detection that learns your data's patterns produces fewer false alarms than static thresholds.

Custom rule support. At some point you'll need business-specific checks. Can you define custom SQL metrics? Can you set validity rules? Can you do referential integrity checks across tables?

Lineage. When something breaks, can you see what's affected downstream? Lineage turns a "this table looks weird" alert into "this table looks weird and it feeds your executive dashboard, the churn model, and the finance report."

Integration with your stack. Alerts should go where your team works (Slack, PagerDuty). The tool should connect to what you already run (dbt, Airflow, Snowflake, Databricks, PostgreSQL). Bonus points for AI agent integration via MCP so your coding assistant can check data health.

The bottom line

Data observability and data quality are complementary, not competing. Observability gives you broad, automated coverage across your entire data estate. Quality gives you precise, rule-based validation on critical data.

If you're starting from zero, start with observability. Connect your databases, get baseline monitoring, and stop finding out about broken pipelines from angry stakeholders. Then add quality checks where they matter most.

If you already have dbt tests or Great Expectations running, you have quality covered. Add observability to catch the problems that explicit tests can't: the pipeline that failed silently, the schema that changed without notice, the table that stopped updating on a holiday.

Either way, the goal is the same: find out about data problems before your stakeholders do.

Data Observability vs Data Quality FAQ

What is data observability?

Data observability is the practice of monitoring data systems end-to-end to understand the health, reliability, and performance of data pipelines. It tracks freshness, volume, schema changes, lineage, and incidents across your data stack. The term is borrowed from software observability but applied to data infrastructure.

What is data quality?

Data quality is the measure of how well data meets the needs of its users. It covers dimensions like accuracy, completeness, consistency, timeliness, uniqueness, and validity. Data quality focuses on the data itself, while observability focuses on the systems producing and moving the data.

Do I need both data observability and data quality?

Most production data teams need both. Observability catches pipeline failures, stale tables, and schema drift. Quality catches bad values, missing records, and business rule violations. They overlap in some areas (freshness, volume anomalies) but diverge in others (lineage vs validation rules). The cleanest approach is to use observability for infrastructure monitoring and quality rules for content validation.

What's the difference between data observability and data monitoring?

Data monitoring is a subset of data observability. Monitoring tracks specific metrics and fires alerts. Observability adds context: lineage showing which pipeline caused a problem, incident history, cross-system correlation, and root cause analysis. Observability is what you do with monitoring data to understand the why, not just the what.

What are the five pillars of data observability?

The five commonly cited pillars are: Freshness (is the data up to date?), Volume (is the expected amount of data arriving?), Schema (has the structure changed?), Lineage (what depends on what?), and Distribution (are the values within expected ranges?). Some vendors add Quality as a sixth pillar.

How does data observability differ from application observability?

Application observability tracks request latency, error rates, and resource usage in services. Data observability tracks data characteristics: freshness, volume, schema, and statistical properties. The underlying principle is the same (instrument everything so you can diagnose problems), but the metrics and tools are different.

What tools provide data observability?

Popular data observability platforms include AnomalyArmor, Monte Carlo, Metaplane, Bigeye, Datafold, Soda, and Databand. Open-source options include Great Expectations, Elementary, and re_data. Each has different strengths in terms of platform support, setup complexity, and price.

Is dbt enough for data quality?

dbt provides tests (schema tests, custom SQL tests) that work for deterministic validation inside your transformation layer. dbt is not enough for production data quality because it doesn't monitor raw source tables, doesn't track freshness across jobs, doesn't provide cross-pipeline lineage, and doesn't detect statistical anomalies. Most teams pair dbt tests with a data observability tool.

How much does data observability cost?

Pricing varies widely. Enterprise tools like Monte Carlo start at $15-25k/year for small deployments. Mid-market tools like Metaplane and AnomalyArmor price per monitored table, typically $5-10/table/month. Open-source tools have no license cost but require engineering time to maintain. Budget based on your number of tables and the criticality of your data.

Can I build data observability in-house?

Yes, but most teams outgrow custom solutions within 6-12 months. In-house data observability typically covers 2-3 pillars well (usually freshness and volume) but falls short on lineage, incident management, and statistical anomaly detection. If you have <20 critical tables and a strong data engineering team, in-house can work. Past that, buying a tool is cheaper than building.

AnomalyArmor combines data observability and quality monitoring in one platform. Try the schema drift demo to see how the AI agent handles both.

Data Quality Monitoring for Snowflake and Databricks: A Practical Guide

Blaine Elliott — Sat, 11 Apr 2026 22:04:59 +0000

Last Tuesday a source team renamed order_status to status in Snowflake. No announcement. Your dbt models kept running, but every query referencing order_status silently returned NULLs. The revenue dashboard showed a 40% drop. It took four hours to trace it back to a one-word column rename.

Schema changes like this are the #1 cause of silent data pipeline failures, and neither Snowflake nor Databricks will warn you when they happen. Data quality monitoring catches these problems automatically. Here's how to set it up for the two most common cloud data platforms.

Four types of data quality monitoring

Data quality monitoring breaks down into four categories:

Schema change detection. A column gets added, removed, renamed, or changes type. Your query doesn't error. It just returns NULLs or wrong values.

Freshness monitoring. A daily table hasn't updated in 36 hours. Something upstream is broken and nobody knows yet.

Anomaly detection. Row count drops 80% overnight. Null rate in a critical column spikes from 2% to 40%. Statistical anomalies surface data quality issues before dashboards break.

Correctness checks. Is order_total ever negative? Does every customer_id in the fact table exist in the dimension table? Business-logic validations specific to your data.

Data quality monitoring in Snowflake

Schema tracking

Snowflake's INFORMATION_SCHEMA tracks table and column metadata:

SELECT table_name, column_name, data_type, ordinal_position
FROM information_schema.columns
WHERE table_schema = 'PUBLIC'
ORDER BY table_name, ordinal_position;

The limitation: this shows current state, not change history. To detect changes, you need to snapshot metadata on a schedule and diff it.

Freshness

INFORMATION_SCHEMA.TABLES.LAST_ALTERED gives you the last DDL or DML timestamp. For row-level freshness:

SELECT MAX(updated_at) as last_data_timestamp
FROM my_schema.my_table;

Where Snowflake's built-in tools fall short

Time Travel provides 90 days of history but querying it at scale for monitoring is expensive.
Streams and Tasks detect changes but require setup per table with no unified view.
Dynamic Tables handle some freshness concerns but don't cover schema changes or anomalies.

Data quality monitoring in Databricks

Schema tracking

Delta Lake enforces schemas by default. DESCRIBE HISTORY shows schema evolution over time:

DESCRIBE HISTORY my_catalog.my_schema.my_table;

This is more useful than Snowflake's current-state-only metadata because you can see what changed and when.

Freshness

Delta table history includes commit timestamps:

DESCRIBE HISTORY my_table LIMIT 1;

Where Databricks' built-in tools fall short

Unity Catalog focuses on access control, not data quality monitoring.
Lakehouse Monitoring (preview) provides profiling but is limited to Databricks-native tables.
Delta Live Tables expectations handle correctness within DLT pipelines but don't cover freshness or schema drift across your broader data estate.

What built-in tools miss

Both platforms provide the primitives: metadata queries, change history, schema enforcement. Turning those primitives into monitoring requires:

Scheduling checks across hundreds of tables every 15-60 minutes
Diffing current state against historical snapshots to detect changes
Per-table thresholds because a real-time events table and a weekly rollup need different SLAs
Alert routing to the team that owns each pipeline, not a shared channel everyone ignores
Cross-platform visibility because most teams run Snowflake and Databricks alongside BigQuery, Redshift, or PostgreSQL. Monitoring each platform separately means blind spots at the boundaries.

Building this yourself is the common starting point. A scheduled SQL job checks freshness, a Python script diffs schemas, a cron sends Slack messages. It works at 20 tables. At 200 tables across three databases, it becomes its own engineering project. (The 2026 State of Data Engineering Survey found that this kind of reactive maintenance consumes 60% of data engineering time.)

This is where dedicated monitoring tools earn their keep. Instead of maintaining a patchwork of scripts, you connect your warehouses once and get schema change detection, freshness monitoring, and anomaly alerts across everything. AnomalyArmor does exactly this for Snowflake, Databricks, BigQuery, Redshift, and PostgreSQL. Email support@anomalyarmor.ai for a trial code.

Getting started

If you're evaluating data quality monitoring for Snowflake or Databricks:

Inventory your critical tables. Which 10-20 tables would cause the most pain if they broke silently?
Set up schema change detection first. Highest value, lowest effort. Schema changes cause silent failures that are expensive to debug after the fact.
Add freshness monitoring for daily-batch tables. The most common source of "the dashboard is showing yesterday's data" incidents.
Layer in anomaly detection once you have baseline data. Anomaly detection needs history to establish normal patterns. Start collecting now.

Snowflake and Databricks Data Quality FAQ

How is data quality monitoring different in Snowflake vs Databricks?

Snowflake exposes metadata through INFORMATION_SCHEMA and ACCOUNT_USAGE, which makes schema tracking and freshness checks straightforward via SQL. Databricks uses Unity Catalog plus Delta Live Tables expectations, which puts data quality checks inside the pipeline itself. Both platforms support custom SQL monitors but have different strengths: Snowflake is better for cross-database monitoring, Databricks is better for streaming and pipeline-level quality gates.

What is schema drift in Snowflake?

Schema drift in Snowflake refers to unexpected changes in table structure: columns added, dropped, renamed, or having their types changed. Snowflake's INFORMATION_SCHEMA.COLUMNS view tracks all column changes via the LAST_ALTERED timestamp, which makes schema drift detection possible with a scheduled query that compares current state against a snapshot.

How do Delta Live Tables expectations work?

Delta Live Tables (DLT) expectations are SQL predicates attached to tables in a pipeline. You write a check like @dlt.expect("valid_id", "id IS NOT NULL") and DLT enforces it on every row. Rows that fail can be dropped, logged, or cause the pipeline to fail. It's data quality enforcement built into the pipeline, rather than monitoring after the fact.

Can I use the same data quality tool for Snowflake and Databricks?

Yes. Tools like AnomalyArmor, Monte Carlo, Metaplane, and Soda work across Snowflake, Databricks, BigQuery, and PostgreSQL. They abstract the warehouse-specific metadata queries behind a unified interface. This is important if you use both platforms, because maintaining separate monitoring systems for each creates operational overhead.

What are Snowflake Data Metric Functions?

Snowflake Data Metric Functions (DMFs) are built-in functions that measure specific data quality attributes: null count, distinct count, duplicate count, freshness, and custom metrics you define. They run as scheduled jobs on a table and store results in a metadata table. DMFs are native to Snowflake but limited to the platform and lack alert routing or cross-table analysis.

How do I monitor data quality across multiple warehouses?

Use a tool that connects to all your warehouses and provides a unified view. Running separate monitoring per warehouse creates blind spots at the boundaries: when data moves from Snowflake to Databricks to a downstream warehouse, you need end-to-end visibility. Cross-warehouse tools handle schema drift tracking, freshness monitoring, and anomaly detection in one place.

What's the cost of data quality monitoring in Snowflake?

Monitoring queries consume Snowflake credits. Schema metadata queries are cheap (milliseconds). Statistical queries on large tables can be expensive if run frequently. Most monitoring tools optimize by sampling, caching, and running queries on off-peak warehouses. Budget 1-5% of your Snowflake spend for monitoring queries if you monitor 50+ critical tables.

Should I use dbt tests or a data observability tool?

dbt tests are good for deterministic validation: "this column should never be null", "these two tables should have matching row counts". Data observability tools are good for statistical anomaly detection, schema drift tracking, freshness monitoring, and cross-table analysis. Most teams use both: dbt tests inside the transformation layer, and observability tools for production monitoring across all data sources.

How do I detect when a Databricks table stops updating?

Query information_schema.tables for the last_altered timestamp, or check the Delta transaction log for the most recent commit. Compare against a freshness SLA. You can also watch for a gap between the expected job schedule and the actual last update time. Dedicated monitoring tools automate this by tracking per-table freshness patterns.

What's the minimum setup time for data quality monitoring?

For a single table with basic checks (row count, null rate, schema), you can set up monitoring in under 10 minutes with most tools. For comprehensive coverage across 50+ tables with alert routing, SLAs, and seasonality detection, budget 1-2 days of configuration. Tools with AI-powered setup (like AnomalyArmor's agent) can configure monitoring from natural language in minutes.

AnomalyArmor monitors Snowflake, Databricks, PostgreSQL, BigQuery, and more from a single interface. See how the AI agent configures monitoring in seconds.

Data Freshness Monitoring: How to Detect Stale Data Before It Breaks Dashboards

Blaine Elliott — Sat, 11 Apr 2026 22:04:45 +0000

A dashboard shows yesterday's revenue as $0. The CEO pings the data team. Someone checks the pipeline, finds the source table hasn't updated in 18 hours, and kicks off a manual backfill. The dashboard was wrong for half a day before anyone noticed.

This is the most common data quality incident in production, and the easiest to prevent. Data freshness monitoring checks whether your tables are updating on schedule and alerts you when they stop.

What is data freshness in data engineering?

Data freshness measures how recently a table or dataset was updated. A table with a freshness SLA of 1 hour should have new data no older than 60 minutes. If it falls behind, something is broken upstream: a failed job, a delayed extract, a stuck queue.

Freshness is different from correctness. A table can be perfectly fresh and full of wrong data. But staleness is the most visible failure mode because dashboards go blank, reports show zeros, and stakeholders notice immediately.

Most data teams discover freshness issues reactively. Someone complains, an engineer investigates, and the root cause turns out to be a pipeline that failed silently hours ago. Industry surveys consistently find that data teams spend over half their time on this kind of reactive work.

Why stale data is expensive

Stale data costs more than the incident itself:

Investigation time. The engineer who gets paged spends 30-60 minutes tracing the staleness back through the pipeline. Which upstream table stopped updating? Which job failed? When?
Trust erosion. Every time a dashboard shows stale numbers, stakeholders trust the data less. Once trust is gone, they build their own spreadsheets and stop using the centralized data platform.
Cascading failures. One stale source table can affect dozens of downstream tables, dashboards, and reports. By the time someone notices, the blast radius has grown.

How data freshness monitoring works

A freshness monitor does three things:

1. Tracks update timestamps. For each monitored table, it records when new data last arrived. This can use metadata queries (information_schema), row timestamps, or partition metadata depending on your warehouse.

2. Compares against a threshold. You define how stale is too stale. A real-time events table might need a 15-minute threshold. A daily aggregate might tolerate 25 hours.

3. Alerts when the threshold is breached. The monitor sends an alert to Slack, email, or PagerDuty. The alert should include which table, how late it is, and ideally what upstream dependency is the likely cause.

How to set data freshness SLAs

The most common mistake is setting the same threshold for every table. A 1-hour SLA on a table that updates weekly creates noise. A 24-hour SLA on a real-time table misses every incident.

Start with how the table is consumed:

Real-time / streaming (event logs, clickstream): 15-30 minute SLA
Hourly batch (hourly aggregates): 2 hour SLA
Daily batch (daily snapshots, dim tables): 25-26 hour SLA
Weekly (weekly rollups): 8 day SLA

Set the SLA slightly longer than the expected update interval. A table that updates every hour should have a 2-hour SLA, not a 61-minute one. This avoids false alerts from minor delays while still catching real failures.

Why pipeline orchestration alerts aren't enough

Airflow, Dagster, and Prefect already alert on job failures. Why add freshness monitoring?

Because jobs can succeed without producing data. A DAG completes with a green checkmark and writes zero rows because the source API returned empty. The orchestrator sees success. The table is stale. Nobody knows until a dashboard breaks.

This is the blind spot that catches most teams. Orchestration monitors the process: did the job run? Freshness monitoring checks the outcome: did data actually arrive? Those are different questions with different answers, and the gap between them is where stale data incidents live.

If you've ever had a "successful" pipeline run that produced an empty table, you've hit this blind spot. Freshness monitoring is the only way to catch it.

Why DIY freshness monitoring breaks down

You can query INFORMATION_SCHEMA.TABLES.LAST_ALTERED in Snowflake, run DESCRIBE HISTORY in Databricks, or check last_modified_time in BigQuery. One table is easy.

The problem is scale. At 50 tables across two databases, you need:

A scheduler running checks every 15-60 minutes
Per-table thresholds (not every table has the same SLA)
Historical tracking to distinguish "late today" from "always late on Sundays"
Alert routing to the team that owns each pipeline, not a shared channel everyone ignores
Cross-database visibility so you can trace a stale table to its upstream root cause in a different warehouse

A cron job with a SQL query gets you started. Maintaining that cron job across hundreds of tables, multiple warehouses, and changing schedules becomes its own engineering project. Most teams that start with DIY eventually spend more time maintaining the monitoring than they saved by catching stale data.

Getting started with data freshness monitoring

If you're setting up freshness monitoring for the first time:

Start with 5-10 critical tables. The ones that power executive dashboards and key reports.
Set generous SLAs initially. Tighten them after you understand normal patterns. Starting too tight creates alert fatigue.
Route alerts to owners. A freshness alert should go to the team that owns the pipeline, not a shared channel where it gets ignored.
Track resolution time. How long between "table went stale" and "table is fresh again"? This tells you if monitoring is actually reducing incident duration.

How AnomalyArmor handles freshness monitoring

AnomalyArmor is built to handle the problems described above so you don't have to build and maintain the monitoring infrastructure yourself.

Automatic pattern detection. When you connect a database, AnomalyArmor runs schema discovery and analyzes historical update patterns for each table. Instead of manually figuring out that orders updates hourly and daily_revenue updates at 6am, the system detects those patterns and suggests appropriate thresholds. You review and adjust, but you're not starting from zero.

Per-table thresholds with SLA tracking. Every table gets its own freshness configuration. Set different thresholds for your real-time event tables (15 minutes) and your weekly rollups (8 days). The freshness chart shows historical update patterns alongside SLA lines so you can see at a glance whether a table is trending toward a violation.

Operating schedules and blackout windows. Not every alert matters at 3am on a Sunday. AnomalyArmor lets you define operating schedules (only alert during business hours) and blackout windows (suppress alerts during planned maintenance). This is the difference between a monitoring tool your team trusts and one they mute.

Cross-warehouse visibility. Monitor Snowflake, Databricks, BigQuery, Redshift, and PostgreSQL from a single dashboard. When a table goes stale, trace the issue upstream across databases using dbt lineage integration to find the root cause, not just the symptom.

Alert routing to the right people. Freshness alerts go to Slack, email, or webhooks. Route different tables to different channels or teams. An alert about a finance table goes to the data platform team. An alert about marketing attribution goes to the analytics team. No shared channel where everything gets ignored.

AI-powered investigation. When a freshness violation fires, AnomalyArmor's AI investigation correlates it with other recent incidents, identifies the likely root cause, and suggests resolution steps. Instead of spending 30 minutes tracing the problem, you get a summary of what broke and why.

Data Freshness Monitoring FAQ

What is data freshness monitoring?

Data freshness monitoring is the process of automatically checking whether tables, datasets, or data streams are updating on their expected schedule. It alerts you when data stops arriving so you can fix the problem before downstream dashboards, reports, or ML models are affected.

How is data freshness different from data quality?

Data freshness measures how recently data was updated. Data quality measures whether the data itself is correct, complete, and consistent. A table can be perfectly fresh but contain wrong values. Freshness is one dimension of data quality, but most teams monitor them separately because the failure modes and fixes are different.

What is a data freshness SLA?

A data freshness SLA is a service level agreement that specifies how often a table should update. For example, "the orders table must have new rows every 1 hour during business hours, or alert." SLAs turn vague expectations into measurable thresholds that monitoring tools can check.

How do I detect stale data in Snowflake?

In Snowflake, query ACCOUNT_USAGE.TABLES for the LAST_ALTERED timestamp on each table, or check the max value of a timestamp column in the data itself. Compare against an expected freshness SLA. If the data is older than the SLA, fire an alert. Most production systems automate this with a scheduled query or a dedicated monitoring tool.

How do I detect stale data in Databricks?

Databricks exposes table modification times through Unity Catalog and Delta Lake transaction logs. Query information_schema.tables or the Delta history to find the last write. Compare against a freshness SLA. You can also use Delta Live Tables expectations to fail pipelines when source data is too old.

Why aren't Airflow (or other orchestration) success alerts enough for freshness monitoring?

Orchestration alerts tell you whether jobs ran successfully. They don't tell you whether the data that arrived is fresh. A job can succeed while producing stale data if the upstream source is delayed, if the extract reads from a cached snapshot, or if the job runs but processes zero new rows. Freshness monitoring checks the data itself, not the job status.

What causes data freshness failures?

The most common causes are: upstream source system outages, failed or delayed scheduled jobs, permission changes that block reads, rate-limited APIs, exhausted compute resources, schema changes that break extract logic, and timezone bugs that misalign expected run times with actual run times.

How often should I check data freshness?

The check frequency should match the SLA. If a table updates hourly, check every 5-15 minutes. If it updates daily, check hourly. Checking too frequently wastes compute. Checking too infrequently means you find failures hours after they happen.

Can I monitor data freshness without a dedicated tool?

Yes, for a small number of tables. Write a scheduled query in Airflow or dbt that checks the max timestamp column and raises on failure. This breaks down at 50+ tables because you need alert routing, incident history, SLA tracking, pattern detection (is this table always late on Mondays?), and snoozing. Dedicated tools handle all of that.

What tools can monitor data freshness?

Popular options include AnomalyArmor, Monte Carlo, Metaplane, Soda, Datafold, and dbt tests (with custom freshness macros). Open-source options include Great Expectations and re_data. Custom scripts work for small teams but don't scale past a handful of tables.

Want to set up freshness monitoring in minutes? Try AnomalyArmor or watch the schema drift demo for a taste of how our AI agent configures monitoring.