Managing data used to mean wrestling with slow queries, complex infrastructure, and the constant fear of crashing a server. For a long time, the systems we used to store and analyze large amounts of information were heavy, rigid, and incredibly difficult to maintain. That all changed with the rise of modern cloud computing and the birth of Snowflake. By completely rethinking how a database should operate, it removed the traditional bottlenecks that kept developers and analysts waiting for hours to get insights. Let’s look at exactly how this platform works, starting with the core definition.
What is snowflake?
Snowflake is a cloud-native SQL platform that decouples storage and compute, allowing all teams to centralize their data pipelines and analytics in one unified ecosystem.
It is a complete sql warehouse
Only built for the cloud
Delivered as a service
Where does Snowflake fits in the Data Landscape?
To understand this better, we have to look at how data landscapes operated in the past.
Legacy data landscape
First, the raw data must pass through a middle processing filter, ETL. This step acts like a cleanup crew, translating, cleaning, and reformatting the messy information from different sources so it all matches a single, standard format.To handle this stage, some data tools are used (like Informatica or Talend) just to move and clean the data.
Once cleaned, this structured information is loaded into a massive central storage system called the EDW (Enterprise Data Warehouse). This central warehouse serves as the main storage vault. AS this is a completely separate layer, it requires an entirely different set of database software tools (like Teradata or Oracle).
Finally, the data is split into smaller, separate databases called Datamarts. This allows individual teams like Finance or Marketing to pull information from their own small buckets to create their final BI / Analytics business reports without breaking the system. Managing these mini-databases requires yet another set of separate database tools.
How does snowflake change this legacy system?
Snowflake completely changes this game by replacing that entire messy puzzle of different tools with one single, unified cloud platform. As you can see in the new image, the separate ETL tools, the heavy EDW hardware, and the isolated physical Datamarts are all gone, swallowed up by the big blue Snowflake box. Instead of juggling multiple expensive software systems, companies can now load all their raw data straight into Snowflake, where it cleans the data, stores it, and shares it with individual teams automatically in one place.
Snowflake's Features
- Complete SQL database
As we said, Snowflake is SQL based warehouse. This means that if you know sql then you already know 80% of snowflake.
- It is a self tuning platform
You don't have to worry eg if your query is not optimized, snowflake detects that and does it for you.
- All the data
Snowflake stores all types of data from all types of sources; structured, semi-structured and unstructured data.
- All your Users
Since data is tored in a centralized location in snowflake, all users are able to access the data.
- Pay Only What you use
Snowflake works on a credit-based system, eg whatever queries you run, you can only pay for that. THere are no hidden costs etc.
- Live data sharing
Since the data is stored in a centralized place, whatever action is done on the data eg Update, every user can be able to see what has changed
Snowflake's Editions
The image below clearly explains this.
Snowflake's Archtecture.
To better understand this architecturelook at the traditional architectures we had before snowflake.
- Shared-disc Traditional Architecture
Imagine a busy office kitchen where multiple chefs (the computers) share one giant refrigerator (the central storage disk). Whenever a chef needs ingredients to cook a meal (process data), they have to walk over to that single refrigerator. While this keeps all the food in one place, it creates a massive bottleneck if all the chefs try to open the refrigerator door at the exact same time, everything slows down to a crawl.
- Shared-nothing Traditional ARchitecture
To fix that bottleneck, companies tried a different setup. This time, every single chef gets their own tiny, private kitchen with their own personal mini-fridge. The chefs no longer fight over one big refrigerator, which makes them faster on their own. However, if Chef A suddenly needs an ingredient locked inside Chef B's mini-fridge, they cannot easily get to it. The systems cannot talk to each other efficiently, forcing teams to duplicate data and work in completely isolated bubbles.
snowflake architecture
Snowflake completely changes the game by combining the absolute best parts of both worlds into what is called a multi-cluster, shared data architecture.Instead of making you choose between the two old methods, Snowflake splits storage and compute into two separate layers.
First, it takes the best part of the shared-disk model by storing all of your company's data in one central cloud vault, ensuring there is only ever one version of the truth.
Then, it borrows the best part of the shared-nothing model by surrounding that central storage vault with multiple, completely independent virtual computers (compute clusters).Because these virtual computers are entirely independent, the marketing team, the finance team, and the data science team can all pull data from the exact same central vault at the exact same moment. They never have to fight for resources, they never slow each other down, and the system never crashes.
Snowflake's Architectural Layers
Snowflake has 3 layers. We will dive into each to better understand them.
Cloud Service Layer
This layer operates as the manager, and brain of the whole snowflake operation.
While the bottom layer stores your data and the middle layer does the heavy math computing, this top layer stands above them to coordinate everything. When a user logs in, writes a query, or requests a report, they never talk to the raw data directly. Instead, they talk to Cloud Services. It takes care of all the background paperwork, security checks, and organization automatically, so your virtual computers can focus entirely on processing data.
This layer does the following operations step by step.
- Management:
This handles the basic operation of the system. It deals with user logins, establishes connections, and manages your active sessions so you can work safely in your web browser.
- Availability:
This ensures your system stays online by automatically syncing your data across three physically separate Availability Zones. Because these data centers are geographically separated and run on independent power grids, a localized blackout that takes out one zone won't affect the others. If a failure occurs, Snowflake instantly and invisibly switches your work to a healthy zone so you never experience downtime or interrupted reports.
- Optimization:
This function acts like a built-in GPS for your data requests. Whenever you type a query, Snowflake’s SQL Optimizer automatically calculates the fastest and cheapest way to find that information without requiring any manual tuning or indexing from the user. It constantly gathers behind-the-scenes statistics and uses metadata to instantly skip past irrelevant data files, ensuring your search results load with maximum speed
- Transactions:
This function acts like a data safety net to ensure your information never gets corrupted or half-broken during changes. It supports full CRUD operations (creating, reading, updating, and deleting data) and follows strict ACID rules, meaning that when you make a massive data update, Snowflake ensures the entire change completes perfectly or none of it does at all. It also handles concurrency management, which keeps the system stable so multiple users can safely modify data at the exact same time without overwriting or locking each other out.
- Security:
This function acts as the ultimate gatekeeper for your entire system, protecting data automatically from the inside out. It keeps all data fully encrypted both while sitting in storage and while moving across networks, making it unreadable to outsiders. It also manages access control, which lets companies set strict user permissions so employees can only see the specific data tables needed for their job roles, ensuring complete privacy.
Storage Layer
This layer is a hybrid columnar structure that works on a concept called micro-partitioning.
What is Micro-Partitioning?
This is the core concept of Snowflake's storage layer, where large tables are automatically chopped up into small, bite-sized units of storage. Each micro-partition holds between 50 to 500 MB of uncompressed data, which Snowflake compresses down to a maximum of 16 MB. These blocks are completely immutable, meaning they cannot be modified once written; any updates simply create new blocks. The Cloud Services layer keeps an inventory of metadata for every single micro-partition—such as the minimum and maximum values inside—allowing Snowflake to instantly target or skip blocks to read your data at lightning speeds.
Micro-Partitioning Example:
As shown in the diagram, when you load a standard table with names like John, Scott, and Mary into Snowflake, the system automatically groups them into small, physical data blocks based on the exact order they arrive. Instead of saving the whole list as one giant file, Snowflake breaks them up into separate segments—putting the first three rows into Partition 1 (P1) and the next three into Partition 2 (P2). Because these files are completely unchangeable (immutable), if you ever need to correct a name or update a record, Snowflake never edits the original file; it simply creates a brand new version of that micro-partition behind the scenes.
Metadata:
This function acts like a super-detailed library catalog for your data blocks, tracking everything without opening the actual files. At the table level, it logs basic stats like the total row count and storage size, while at the micro-partition column level, it records the exact range of data inside, including minimum, maximum, and empty (NULL) values. As shown in the diagram, Snowflake instantly knows that names starting with 'J' through 'S' are in the first partition, allowing it to instantly skip the second partition when you search for "John" and deliver your results at lightning speed.
Compute Layer
This layer acts as the muscle of Snowflake, consisting of independent groups of virtual computers in the cloud called Virtual Warehouses that do all the heavy data processing. As the slide shows, these warehouses come in sizes ranging from Extra Small (XS) to Extra Large (XL) and beyond, where going up a size simply doubles the number of servers in the cluster to give you more speed. You only pay for these servers while they are actively running, and because they are completely separate from each other, multiple departments can process data at the same time without ever slowing down each other's work.
Workload Segmentation:
This feature prevents different business operations from fighting over the same computer power by creating dedicated virtual warehouses for specific tasks. As shown in the diagram, instead of sharing one system, you can assign completely separate virtual computers to individual workloads like heavy ETL cleanup, continuous data loading, interactive business dashboards, and team reporting. Because these compute clusters are completely isolated from each other, a massive data loading job running in the background will never cause a business executive's live dashboard to slow down or lag.
Scale up & Scale Out Concepts in Snowflake
Scaling Up:
This is the process of increasing the size of a single virtual warehouse to give you a massive boost in raw processing power. As the slide illustrates, if you have a highly complex calculation or a giant dataset to ingest, you can instantly change your warehouse size, like upgrading from a Medium to a Large which doubles the computing resources (CPU, RAM, and SSD) available for that specific task. Because Snowflake uses per-second billing, scaling up allows you to finish massive, complex jobs much faster for roughly the same cost, though it is strictly designed for handling large workloads rather than managing traffic from a high number of simultaneous users.
Scaling Out:
This is the process of automatically adding identical computer clusters to a single virtual warehouse to handle high user traffic, known as concurrency. As the slide outlines, instead of making a single computer bigger, scaling out automatically boots up extra, side-by-side computer clusters during peak rush hours to evenly balance the incoming query load. Snowflake dynamically adds these clusters when user demand spikes and shuts them down when things slow down, using simple settings (MIN and MAX cluster counts) to ensure your dashboards never lag or freeze when hundreds of employees log in at the exact same time.
In Part 2 of this series, we will jump directly into the Snowflake platform itself. We will break down how Snowflake handles security using Users and Roles, look at how data is organized into Databases and Schemas, and explore the different table types you can use to build your very first data pipeline.
Top comments (0)