I'm starting a new learning journey: Apache Spark.
Rather than simply watching tutorials and memorizing Spark APIs, I want to understand what Spark is actually solving, how it works internally, and why it has become such an important tool in data engineering.
So this series will be my attempt to learn Spark in public.
I'm approaching each topic from a simple perspective:
Understand the problem first. Then understand the technology that solves it.
And before writing my first Spark program, I think there's one important question to answer:
Why does Apache Spark exist in the first place?
What Is "Big Data"?
You've probably heard the term Big Data countless times.
A simple definition is:
Data becomes "big" when its volume, velocity, variety, or processing requirements exceed what traditional systems can handle efficiently and economically.
Notice that this doesn't necessarily mean:
"The data is larger than RAM."
That's only one possible limitation.
For example, a 2 TB dataset could theoretically be processed by a machine with enough memory. But buying and maintaining a machine with enormous amounts of RAM may not be practical or economical.
The real question is:
When does processing data on a single machine stop being practical?
That's where distributed computing becomes interesting.
The 5 Vs of Big Data
A common way of describing Big Data is through the 5 Vs.
| V | Meaning | Example |
|---|---|---|
| Volume | Amount of data | Terabytes of logs |
| Velocity | Speed at which data is generated | Millions of events per minute |
| Variety | Different types and formats | JSON, CSV, logs, images |
| Veracity | Data quality and reliability | Missing or inconsistent data |
| Value | Useful information extracted from data | Customer insights |
Not every data problem involves all five.
But these dimensions help explain why traditional approaches can eventually become difficult to scale.
Why Can't We Just Use Pandas?
Let's start with something familiar.
Suppose we have a sales dataset:
import pandas as pd
df = pd.read_csv("sales.csv")
result = df.groupby("customer")["amount"].sum()
For a reasonably sized dataset, this is perfectly fine.
Pandas is an excellent tool for:
- Data analysis
- Data cleaning
- Exploratory data analysis
- Prototyping
- Working with small and medium-sized datasets
But now imagine that the dataset is 500 GB.
Your laptop might have:
CPU: 8 cores
RAM: 16 GB
Trying to load hundreds of gigabytes into memory on that machine isn't practical.
And there's another important detail.
A CSV file's size on disk isn't necessarily the same as the memory required by the resulting DataFrame.
Pandas needs memory for:
- Data values
- Column structures
- Indexes
- Strings
- Internal data structures
- Other memory overhead
So a 10 GB CSV can require significantly more than 10 GB of RAM when loaded into memory.
Eventually, the single-machine approach reaches a limit.
The Single-Machine Problem
Imagine we have:
10 TB of data
but our server has:
64 GB RAM
We can't simply load the entire dataset into memory.
One option would be to buy a much larger machine.
That brings us to vertical scaling.
Vertical Scaling
Vertical scaling means making one machine more powerful.
For example:
Before
┌───────────────────┐
│ Server │
│ │
│ 8 CPU cores │
│ 32 GB RAM │
└───────────────────┘
↓
After
┌───────────────────┐
│ Server │
│ │
│ 64 CPU cores │
│ 256 GB RAM │
└───────────────────┘
This can work very well for many workloads.
But it has limitations.
Hardware becomes increasingly expensive, and eventually there are physical and economic limits to how large a single machine can become.
So we have another option.
Horizontal Scaling
Instead of buying one enormous machine, we can use multiple machines.
For example:
10 TB Dataset
│
┌────────────┼────────────┐
↓ ↓ ↓
Server 1 Server 2 Server 3
3.3 TB 3.3 TB 3.4 TB
Each machine can process part of the dataset.
This is horizontal scaling.
Instead of:
One extremely powerful machine
we use:
Many machines working together
This leads us to an important concept named distributed computing.
What Is Distributed Computing?
Distributed computing is an approach where multiple computers work together over a network to solve a problem by dividing the workload among them.
For example:
Dataset
│
┌─────────┼─────────┐
↓ ↓ ↓
Worker 1 Worker 2 Worker 3
│ │ │
Part A Part B Part C
│ │ │
└─────────┼─────────┘
↓
Combined Result
Instead of one machine processing everything, different machines process different portions of the workload.
And because they can work in parallel, the total processing time can potentially be reduced significantly.
But Distributed Computing Isn't Free
At first, distributed computing sounds simple:
"Just add more machines."
Unfortunately, it isn't that easy.
Imagine a cluster containing hundreds of machines.
Now we have new problems.
What if a machine fails?
Worker 1 ✓
Worker 2 ✓
Worker 3 ✗
Worker 4 ✓
What happens to the work assigned to Worker 3?
How do we divide the data?
We need to decide which machine processes which portion of the dataset.
How do machines communicate?
The machines need to exchange information over a network.
Network communication isn't free.
How do we combine the results?
Each machine produces partial results.
We eventually need to combine them into the final result.
What if one machine is much slower?
Suppose:
Worker 1 → 10 seconds
Worker 2 → 12 seconds
Worker 3 → 11 seconds
Worker 4 → 3 minutes
The entire operation may have to wait for the slow worker.
This is one of the challenges of distributed systems.
So We Have a New Problem
We've gone from:
Single Machine
│
↓
Data becomes too large
│
↓
Add more machines
│
↓
Distributed Computing
│
↓
Now we need to coordinate
│
↓
Distributed Processing Frameworks
And this is where technologies such as Hadoop and later Apache Spark come into the picture.
Where Does Apache Spark Fit?
At a very high level:
Large Dataset
│
↓
┌─────────────────────┐
│ Distributed │
│ Processing │
└──────────┬──────────┘
│
┌────────┴────────┐
↓ ↓
Hadoop Spark
Hadoop introduced a widely used ecosystem for distributed storage and processing.
Spark later became popular as a powerful distributed computing engine capable of handling a wide variety of workloads.
We'll explore the history and differences in the next article.
An Important Clarification
It's easy to misunderstand this topic and conclude:
"Pandas is bad and Spark is good."
That's not the right way to think about it.
Pandas and Spark solve different problems.
For example:
Pandas
↓
Single-machine data analysis
while:
Spark
↓
Distributed data processing
And Spark isn't automatically better just because the dataset is large.
If you're processing a small dataset on your laptop, using Spark could introduce unnecessary complexity.
The right question is:
What processing architecture is appropriate for this workload?
That's an important mindset for a data engineer.
My First Hands-On Experiment
To understand this problem instead of simply reading about it, I created a CSV file containing 1 million rows and measured how long Pandas took to load it.
The experiment was simple:
Generate CSV
↓
Measure file size
↓
Check system RAM
↓
Load with Pandas
↓
Measure loading time
The code looked roughly like this:
import csv
import os
import time
import pandas as pd
import psutil
file_name = "millions_of_rows.csv"
total_rows = 1_000_000
start_time = time.perf_counter()
with open(file_name, "w", newline="") as file:
writer = csv.writer(file)
writer.writerow(["id", "name", "score"])
for i in range(1, total_rows + 1):
writer.writerow([
i,
f"Person_{i}",
i % 100
])
creation_time = time.perf_counter() - start_time
print(f"CSV created in {creation_time:.2f} seconds")
file_size_bytes = os.path.getsize(file_name)
file_size_mb = file_size_bytes / (1024 ** 2)
ram = psutil.virtual_memory()
total_ram_gb = ram.total / (1024 ** 3)
available_ram_gb = ram.available / (1024 ** 3)
print(f"File size: {file_size_mb:.2f} MB")
print(f"Total RAM: {total_ram_gb:.2f} GB")
print(f"Available RAM: {available_ram_gb:.2f} GB")
print("Reading CSV using Pandas")
start_time = time.perf_counter()
df = pd.read_csv(file_name)
loading_time = time.perf_counter() - start_time
print(f"Rows loaded: {len(df):,}")
print(f"Columns loaded: {len(df.columns)}")
print(f"Loading time: {loading_time:.2f} seconds")
The purpose wasn't to prove that Pandas is slow.
The purpose was to observe what happens as the amount of data increases.
A useful follow-up experiment would be to compare:
100K rows
500K rows
1M rows
5M rows
10M rows
and record:
- File size
- Loading time
- Memory consumption
This gives a much better intuition for the single-machine limitation.
The Bigger Picture
At this point, the evolution should look something like this:
Data
│
↓
Single-machine tools
┌─────────┴─────────┐
│ │
Pandas Databases
│ │
└─────────┬─────────┘
│
↓
Data gets larger
│
↓
Single machine becomes
less practical
│
↓
Horizontal Scaling
│
↓
Distributed Computing
│
↓
Hadoop / Spark
This is the foundation we need before learning Spark.
What I Learned
The biggest takeaway from this lesson for me was:
Spark isn't the starting point of the story. Distributed computing is.
Before learning Spark APIs, it's important to understand why distributing computation across multiple machines becomes necessary in the first place.
The key concepts I want to remember are:
1. Big Data
Data becomes challenging when its scale or processing requirements exceed what traditional approaches can handle efficiently.
2. Vertical Scaling
Make one machine more powerful.
3. Horizontal Scaling
Add more machines.
4. Distributed Computing
Multiple machines work together to solve a problem.
5. The Trade-off
Distributed systems provide scalability, but introduce complexity around:
- Communication
- Network transfer
- Failure handling
- Data partitioning
- Coordination
- Combining results
And this is exactly the problem that distributed processing frameworks are designed to help solve.
What's Next?
Now that we understand why distributed computing is necessary, we can look at the technology that popularized large-scale distributed data processing:
Apache Hadoop & MapReduce
In the next article, I'll explore:
- What Hadoop is
- Why Hadoop was created
- What HDFS does
- What MapReduce is
- Map vs Reduce
- The Shuffle phase
- How a distributed job actually executes
- Why MapReduce can become slow
- And finally...
Why Apache Spark was created as a better approach for many workloads.
Follow the Series
This is Part 1 of my Apache Spark learning journey.
I'm learning Spark from the ground up and documenting what I learn along the way-not as an expert, but as a data engineer trying to understand distributed data processing properly.
If you're also learning Spark, feel free to follow along.
Next: Apache Spark for Beginners #2 - Hadoop, MapReduce, and the Problem Spark Solves
Top comments (0)