It was 3:14 AM on a Tuesday when the PagerDuty alert fired. Our DLT pipeline, processing HIPAA-sensitive patient telemetry, had stalled. A library conflict in a custom Python UDF—introduced via a transient cluster restart—caused a silent data drift in our gold-layer aggregations. By the time we caught the stale dashboards, we’d pushed downstream reports to three hospital systems with flawed occupancy metrics. The cost wasn’t just the $4,000 in wasted DBU consumption during the retry storm; it was the two weeks of manual reconciliation to regain the trust of our compliance team.
Why I chose this topic: I’ve spent the last six years cleaning up broken pipeline architectures in regulated environments. I’m tired of reading marketing fluff that ignores the reality of debugging distributed state; here is how these tools actually behave when the pressure is on.
You are currently deciding how to build your next data platform. You have two primary paths: letting Snowflake handle the state management of your transformations via Dynamic Tables, or letting Databricks manage your DAGs via Delta Live Tables (DLT). Both promise to replace the "if-this-then-that" imperative spaghetti of Airflow with declarative SQL. Both lie about how easy that is.
The contenders
Snowflake Dynamic Tables (DTs) are a logical extension of the Snowflake storage engine. You define a TARGET_LAG (e.g., '1 minute'), and the underlying Snowflake scheduler handles the incremental materialization. It’s essentially a materialized view that doesn’t require you to manually write the refresh logic.
Databricks DLT is an abstraction layer on top of Spark. It manages the infrastructure (clusters), the dependencies, and the data quality (Expectations). It’s a full-fledged orchestration engine hidden inside a SQL/Python framework.
Photo by Logan Voss on Unsplash
The cost of abstracting the scheduler
Snowflake’s DT pricing is straightforward but dangerous. You pay for the compute used during the refresh. If your TARGET_LAG is set to 1 minute, Snowflake’s scheduler is constantly spinning up or using a warehouse to check for changes. If your underlying base table is constantly being ingested into (e.g., streaming Kafka data), that warehouse is running 24/7. I’ve seen junior devs set TARGET_LAG = '1 minute' on a table with 50 terabytes of data, effectively creating a permanent, expensive compute sink.
Databricks DLT pricing is more granular but significantly more complex. You have the DLT compute cost (DBUs) plus the underlying cluster cost. Because DLT uses Spark, it’s remarkably efficient at processing massive, partitioned shuffles—things that would make a Snowflake warehouse cry. However, the "Enhanced Autoscaling" feature is a trap. I’ve seen it spin up an r6id.4xlarge cluster to process a tiny backfill, then hang around for the 15-minute "down-scaling delay," burning money while doing absolutely nothing.
Failure modes and observability
When a Snowflake Dynamic Table fails, you check DYNAMIC_TABLE_REFRESH_HISTORY. The error codes are generally readable: TABLE_NOT_FOUND or COMPUTE_RESOURCE_LIMIT. It’s essentially a black box. If the refresh fails, Snowflake retries based on its own internal logic. You have very little control over the retry backoff or the sequence of operations. It’s "set it and forget it," until it breaks, at which point you are at the mercy of Snowflake Support.
DLT failure modes are far more "interesting." Because DLT is Spark, you get the full, ugly stack trace. You’ll see AnalysisException, Py4JJavaError, and the classic ExecutorLostFailure. The good news is that DLT's EXPECTATIONS are a game-changer. You can define CONSTRAINT patient_id_not_null EXPECT (patient_id IS NOT NULL) ON VIOLATION DROP ROW. This is declarative data quality that actually works. If you try to do this in Snowflake, you’re writing CASE WHEN statements in your select list like it’s 2012.
Photo by Craftsman Concrete Floors on Unsplash
Operational overhead and infrastructure
Snowflake wins on pure platform integration. If your data is already in Snowflake, you don't provision anything. You don't manage cluster sizes. You don't worry about node instance types. You enable the DYNAMIC_TABLE_USER role, grant the permissions, and write your CREATE DYNAMIC TABLE. It is the "I don't have a dedicated DevOps team" solution.
Databricks DLT requires you to be a Spark operator. You need to understand spark.sql.shuffle.partitions, memory overhead, and how to configure your pipelines.yml to ensure your clusters aren't thrashing. If you have a team that knows how to tune a Spark session, DLT is a Ferrari. If you don’t, it’s a Ferrari with the wheels welded on crooked.
What I'd pick, and why
If I’m building a greenfield project for a startup or a mid-sized data team where the primary goal is velocity and standard transformation (JOINs, simple aggregations), I am picking Snowflake Dynamic Tables. The maintenance burden is zero. You can put your entire pipeline in a single schema.sql file and version control it. The lack of granular control is a feature, not a bug, because it prevents your engineers from over-engineering the shuffle partitions.
However, if I am working in healthcare or finance where complex window functions, high-volume streaming, and rigorous data quality constraints are non-negotiable, I am picking Databricks DLT.
The caveat: You must accept that DLT requires a "Platform Engineer" on staff. You cannot treat DLT as a managed service that requires zero intervention. You will spend time monitoring cluster health, tuning storage-to-compute ratios, and debugging the occasional Spark driver OOM (Out of Memory) error.
If you choose Snowflake and your data volume hits the petabyte scale, you will eventually hit the wall of "warehouse compute limits" and move to DLT anyway. Choose Snowflake for the simplicity, but keep your DLT migration plan in a drawer. You’ll need it sooner than you think.
Cover photo by Liam Briese on Unsplash.
Top comments (0)