TL;DR — Snowflake bet that a warehouse rebuilt cloud-native — storage fully separated from compute, no indexes, no knobs — would beat anything retrofitted from on-prem. Databricks bet that the data should stay in open files on cheap object storage and the engine should come to it. Both bets worked, and then both companies spent a decade building the thing they had originally argued against.
Yesterday we left two founding teams who barely shared a vocabulary: Oracle kernel engineers in a San Mateo stealth office writing a warehouse from a blank sheet, and a Berkeley lab shipping a research engine that had just become a top-level Apache project. Today we look at what they actually built, because the fork they took in 2012 and 2013 explains almost every move either company has made since — the acquisitions, the format war, the pricing, even whose vocabulary the industry now speaks.
This is the part of the rivalry where the marketing is loudest and the substance is most checkable. So we will do it the boring way: what each design makes cheap, what it makes expensive, and where each side quietly conceded.
The concepts you need
Seven ideas carry this argument.
Separation of storage and compute. Storage holds bytes; compute is the CPUs reading them. Coupled, you buy disks to get cores and every workload fights the rest. Separated, storage grows on its own, compute suspends when idle, and many clusters read one copy.
Columnar storage and data skipping. Analytics reads few columns over many rows, so data is stored column-by-column: better compression, less I/O. Instead of indexes each file carries min/max stats — zone maps — so the engine skips files that cannot match your filter. Free to maintain, useless if the data is not ordered roughly the way you query it.
A table format. Parquet files in a bucket are not a table: nothing records which files belong, what the schema is, or which version you are reading. Delta, Iceberg and Hudi are metadata logs beside the files that answer that.
ACID on object storage. S3 offers PUT and GET, not transactions. The log restores them — writers append a version, readers pin one, a crashed job leaves orphans no version points at — giving atomic writes, snapshot isolation and time travel.
What a query engine does. It turns a request into a plan and runs it. Spark compiles a DAG of stages across a cluster and also runs arbitrary Python or Scala; a SQL warehouse takes SQL only and hides the plan.
Serverless vs provisioned compute. Provisioned: you size the cluster and pay while it runs. Serverless: the vendor keeps capacity warm, your query lands in seconds, you pay by usage — deleting sizing decisions, idle waste, and your view of what a query cost.
What a catalog governs. Which tables exist, who may read which columns, where data came from. When several engines touch the same files, it is the only place a rule gets enforced once instead of four times.
Snowflake's bet: separate storage and compute, then hide everything
Benoît Dageville has described the founding decision as separating compute from storage "fully and without compromise," which he credits with eliminating resource contention as a limiting factor. The formal version is the team's peer-reviewed SIGMOD 2016 paper, The Snowflake Elastic Data Warehouse, which describes the multi-cluster shared-data architecture: one logical copy of the data, many independent compute clusters reading it.
Concretely, at general availability on AWS in mid-2015 — sources disagree on whether the exact date was June or July — InfoQ reported data stored in compressed 16MB files on S3, pulled into EC2 compute clusters on demand, with zone maps and data skipping in place of traditional indexes. That last detail is the one dev teams feel. There is no CREATE INDEX to get wrong: the engine keeps min/max metadata per file (today marketed as micro-partitions) and skips the ones that cannot contain your predicate. Pruning replaces tuning.
Layer a "virtual warehouse" on top — a named, resizable compute cluster you spin up, scale, and suspend independently — and you get the property that sold the product: your ETL job cannot slow down the CFO's dashboard, because they are not sharing a machine.
The cost of that elegance was the format. Snowflake's optimizations lived inside a proprietary storage layout nothing else could read. That was simultaneously the original sin and the original superpower — the reason the system could be that fast with that few knobs, and the reason every CIO eventually asked what happens to the data if we leave.
Databricks' bet: leave the data where it is, open
Databricks came from the opposite direction. Spark started in 2009 as a Berkeley AMPLab project, built as a pilot workload for the Mesos cluster manager; the HotCloud 2010 paper showed roughly 10x speedups over Hadoop on iterative ML jobs, and the NSDI 2012 RDD paper — a USENIX Best Paper — reported up to 100x over prior multi-pass models. That lineage matters architecturally: Spark was an engine, not a database. It read whatever files you pointed it at.
The problem with pointing an engine at a data lake was that a lake had no transactions. Two writers, one failed job, and you had a half-written table nobody could trust. Delta Lake, open-sourced April 24, 2019 and moved to Linux Foundation governance that October, fixed exactly that: ACID transactions, schema enforcement, and time travel, implemented as a metadata log sitting beside Parquet files. Delta Lake 2.0, announced June 28, 2022, open-sourced the APIs Databricks had kept proprietary until then.
That is the lakehouse thesis in one sentence: keep the cheap open files, add the transaction log, and you no longer need a separate warehouse to get warehouse guarantees. Add a vectorized execution engine (Photon) so SQL on those files is fast, and a governance layer (Unity Catalog) so the files have permissions and lineage, and you have one platform serving BI and ML from one copy of the data — because the alternative is copying everything twice and reconciling it forever.
The head-to-head
| Dimension | Snowflake (2012 bet) | Databricks (2013 bet) |
|---|---|---|
| Storage format | Proprietary columnar files (16MB compressed on S3 at 2015 GA), managed by the service | Open Parquet on your object storage, made transactional by the Delta log |
| Compute model | Virtual warehouses — named, sized, suspendable SQL clusters over one shared data copy | Spark clusters running arbitrary code (SQL, Python, Scala, ML); SQL warehouses came later |
| Elasticity | Resize or add clusters per workload; per-second billing; contention designed out | Scale clusters per job; autoscaling, then serverless SQL warehouses |
| Query optimization | Zone maps and data skipping instead of indexes; no tuning surface | Engine-side vectorized execution (Photon); layout and compaction handled by the table format |
| Governance | In the service from day one — one system, one permission model | Unity Catalog, added later, open-sourced June 12, 2024 with Iceberg REST and Hive metastore compatibility |
| ML / AI story | SQL-first; Python and Spark-shaped workloads arrived later via Snowpark | Native from birth — an ML engine that grew a warehouse |
| Openness | Closed format originally; Iceberg tables and Polaris came later | Open formats as the founding argument; Delta fully open by 2.0 (June 2022) |
| Ops burden | Very low — fewest moving parts here | Higher: more power, more surface, more to get wrong |
Read that table honestly and the trade is obvious. Snowflake sells you fewer decisions. Databricks sells you more control and the right to walk away with your files.
Then both of them defected
The interesting part is not the fork. It is that each company spent the following years building the other's starting point.
| Move | Company | Date | What it concedes |
|---|---|---|---|
| Delta Lake open-sourced | Databricks | Apr 24, 2019 | Lakes needed warehouse guarantees |
| TPC-DS 100TB record set on Databricks SQL | Databricks | Nov 2, 2021 | It wanted to be judged as a warehouse |
| Delta Lake 3.0 with UniForm | Databricks | Jun 28, 2023 | Lock-in is unsellable — Iceberg/Hudi metadata written alongside Delta |
| Polaris Catalog (open Iceberg REST catalog) | Snowflake | Jun 3, 2024 | Open table formats won the argument |
| Unity Catalog open-sourced | Databricks | Jun 12, 2024 | Governance is a standard, not a moat |
| Neon (serverless Postgres) bought, ~$1B | Databricks | May 2025 | The lakehouse needed an operational database |
| Crunchy Data bought, $164.5M cash per SEC filing (announced at about $250M) | Snowflake | Jun 2025 | So did the warehouse |
| Native Iceberg support | Databricks | Jun 2025 | One format to interoperate on |
Snowflake added Snowpark so Python and Spark-shaped workloads could run inside the warehouse, then external and Iceberg tables so data no longer had to live in its format to be queried by it, then Unistore-branded transactional storage — and finally bought a Postgres company outright. Databricks added Databricks SQL and serverless warehouses so BI tools could hit the lakehouse without anyone learning Spark, and bought a serverless Postgres company for roughly $1 billion in the same window.
By 2026 the two architectures rhyme: object storage underneath, open Iceberg-compatible metadata in the middle, serverless SQL on top, a governance catalog beside it, an OLTP Postgres bolted on for agent workloads, and consumption pricing on all of it.
Who actually moved further
Convergence is not symmetry. Measure it by whose vocabulary the industry speaks and whose format it writes.
The Apache Iceberg v3 spec was ratified in 2025 — deletion vectors, a variant type for semi-structured data, row lineage, geospatial types, nanosecond timestamps — and by 2026 both vendors read and write Iceberg natively, with the major clouds shipping managed Iceberg services. Nobody is building a new closed table format, and "lakehouse" is now a category rather than a Databricks slogan.
Databricks played the format endgame ruthlessly: on June 4, 2024, during Snowflake Summit week, it announced the acquisition of Tabular — the company founded by Iceberg's original Netflix creators — after a bidding war that CEO Ali Ghodsi confirmed included Snowflake and Confluent. Price reporting never settled: June 2024 coverage put it near $1 billion, largely performance-based, with Ghodsi indicating near-term cash "in the hundreds of millions," while Bloomberg reporting relayed that August put it closer to $2 billion. Treat that number as disputed.
The honest counterweight: Snowflake's Polaris was open-sourced under Apache 2.0 within about 90 days, donated to the ASF in August 2024, and graduated to a top-level Apache project on February 18, 2026. On catalog openness, the company with the proprietary origin story shipped the more open artifact.
The two stacks today: advantages and disadvantages
Architecture arguments are free; running the thing is not.
Databricks — Spark + Photon, Delta Lake, Unity Catalog, notebooks-first
- Pros. Data stays in open Parquet in your bucket, readable by another engine tomorrow. One platform serves BI, streaming and training from a single copy, removing the copy-and-reconcile tax. You can drop from SQL into Python mid-pipeline. Governance is an open spec — Unity Catalog was open-sourced June 12, 2024, with over 10,000 enterprises reported on the managed version at announcement.
- Cons. Power costs operations: sizing, autoscaling, compaction and layout are yours to get right, and wrong looks like slow jobs and burnt spend. Spark's execution model leaks into your debugging. And an open format is not a painless exit — notebooks, tuning and catalog wiring still have to be unwound.
Snowflake — virtual warehouses, micro-partitions, Snowpark, SQL-first
- Pros. Near-zero ops: no indexes to design, no cluster philosophy, no compaction jobs. Elasticity is per-workload and immediate, so ETL stops competing with the dashboard. Anyone who writes SQL is productive on day one — a hiring advantage, not just a comfort. One governance model from the start, not a retrofit.
- Cons. Proprietary gravity: the fast path is still Snowflake's own storage and functions, and Iceberg tables are a concession, not the default habit. ML is less native — Snowpark arrived years later and heavy training usually leaves the building. And the abstraction that removes knobs removes visibility: with consumption pricing and no plan to inspect, a surprising bill is easy to make and hard to attribute.
They fail in opposite directions: Databricks fails at effort, Snowflake at control.
What to Learn From This
- Ask what an architecture makes expensive, not what it makes fast. Skipping instead of indexing buys zero tuning and bills you the day your queries stop matching your data layout.
- Storage format is your exit clause. "Can something else read this table?" is a procurement question, not a purity argument.
- Count your copies. Most warehouse-plus-lake pain is not query speed; it is the second copy and the reconciliation job nobody owns.
- Match the stack to the team you have. Three analysts and no platform engineer will ship more on the stack with fewer knobs, whoever won the architecture debate.
- Watch what vendors buy, not what they announce. Both bought a Postgres company within a month in 2025, which showed where agent workloads were heading before either roadmap said so.
- Serverless moves cost from your ops budget to your invoice. The work changes department rather than disappearing: budget for spend observability as you would have for tuning.
Scorecard
Round 2: Databricks. The industry now argues in Databricks' terms — lakehouse, open table format, one copy of the data for BI and ML — and Snowflake's own roadmap ratifies it: Snowpark for non-SQL workloads, external and Iceberg tables so data need not live in its format, and an Iceberg REST catalog it built and then gave away. Databricks conceded less of its founding thesis and more of its ergonomics; Snowflake conceded the thesis. But the round is not a rout, and the honest half is this: Snowflake still wins simplicity and operations decisively. No indexes to tune, no cluster sizing philosophy, no governance retrofit, one permission model since day one — for a team that wants dashboards and not a platform, that is worth more than architectural purity. Databricks won the argument; Snowflake still wins the Tuesday afternoon.
Running tally: Databricks 1 — Snowflake 0 (1 draw)
Tomorrow
Part 3 follows the money: the biggest software IPO on record versus the private mega-round machine that overtook it without ever ringing the bell.
Sources
- The Snowflake Elastic Data Warehouse (SIGMOD 2016)
- InfoQ: Snowflake cloud data warehouse GA architecture, 2015
- Resilient Distributed Datasets (USENIX NSDI 2012, Best Paper)
- VentureBeat: Databricks open-sources Delta Lake (Apr 24, 2019)
- PR Newswire: Delta Lake 3.0 and UniForm (Jun 28, 2023)
- Business Wire: Snowflake unveils Polaris Catalog (Jun 3, 2024)
- Databricks: open-sourcing Unity Catalog (Jun 12, 2024)
- CNBC: Databricks acquires Tabular (Jun 4, 2024)
Top comments (0)