Classic data warehouses tie your data to the machines that query it. Add more compute and you have to reshuffle the data. Run a heavy report and it starves your loading jobs. Snowflake's central idea is to break that tie: keep the data in one shared place and let any number of independent compute clusters read it. That one decision reshapes cost, scaling, and concurrency.
The core problem
In a traditional shared-nothing warehouse, each node owns a slice of the data on its local disk. This is fast, because compute sits next to its data. But it couples two things that have nothing to do with each other. Storage grows with how much data you keep. Compute grows with how many queries you run and how fast you want them. Coupling them means every time one dimension changes, the other has to be re-provisioned, and rebalancing data across nodes is slow and disruptive.
Worse, workloads fight. Your BI dashboards, your data science team, and your nightly load all hit the same cluster. One expensive query drags everyone down. You end up over-provisioning for the peak and paying for idle capacity the rest of the time.
Key design decisions
Three layers instead of one. Snowflake splits into storage, compute, and a cloud services layer. Storage is object storage in the cloud, holding table data as compressed files. Compute is made of clusters called virtual warehouses, each a group of nodes that read from that shared storage. The services layer handles metadata, query planning, transactions, and security.
Data lives in object storage, not on compute nodes. Tables are stored as immutable micro-partitions, columnar files of a bounded size. Because the files never change in place, a write produces new files and the metadata layer tracks which files make up the current table. This is what makes time travel and cheap cloning possible: you keep old file sets around and point at them.
Compute is elastic and isolated per workload. Each team spins up its own virtual warehouse. They all see the same data because they read the same storage, but they do not share CPU or memory, so one team's heavy query does not slow another. A warehouse can resize on demand and can be suspended when idle so you stop paying for it. Billing follows compute time, not a fixed cluster.
Metadata makes queries skip data. The services layer keeps min and max values and other statistics per micro-partition. A query with a filter can prune whole files without reading them, which cuts scan cost sharply on selective queries.
The trade-offs
Separating storage and compute adds network distance. Compute no longer sits on top of local disk, so reads cross the network to object storage. Snowflake hides this with aggressive local caching on the warehouse nodes, so hot data stays close, but a cold query pays the fetch cost. You trade the guaranteed locality of shared-nothing for elasticity and isolation.
Immutable storage trades write-in-place efficiency for versioning and concurrency. Updating a few rows means rewriting the affected files, which is heavier than an in-place update. In return you get snapshot isolation, easy rollback, and zero-copy clones, because everything is just pointers to file sets.
Elastic per-team compute can also surprise you on cost. It is easy to leave warehouses running or to scale up more than a query needs. The system bills by compute time, so the flexibility that saves money when managed can waste it when ignored.
How the real system does it
The real architecture is exactly this separation: durable columnar micro-partitions in cloud object storage, independent virtual warehouses that scale and suspend on their own, and a shared services layer that owns metadata, planning, and transactions. Caching on compute nodes recovers most of the locality lost by moving data off local disk, and partition pruning keeps scans cheap.
The general lesson: when two resources scale for different reasons, do not force them onto the same hardware. Decoupling storage from compute is why the warehouse can grow reads and writes independently and bill only for what runs.
I wrote the full breakdown, with diagrams and the data model, here: https://www.systemdesign.academy/interview/design-snowflake
Top comments (0)