Apache Spark for Beginners #1: Why Does Spark Exist?

Sanjeev Kumar — Mon, 10 Aug 2026 14:54:25 +0000

I'm starting a new learning journey: Apache Spark.

Rather than simply watching tutorials and memorizing Spark APIs, I want to understand what Spark is actually solving, how it works internally, and why it has become such an important tool in data engineering.

So this series will be my attempt to learn Spark in public.

I'm approaching each topic from a simple perspective:

Understand the problem first. Then understand the technology that solves it.

And before writing my first Spark program, I think there's one important question to answer:

Why does Apache Spark exist in the first place?

What Is "Big Data"?

You've probably heard the term Big Data countless times.

A simple definition is:

Data becomes "big" when its volume, velocity, variety, or processing requirements exceed what traditional systems can handle efficiently and economically.

Notice that this doesn't necessarily mean:

"The data is larger than RAM."

That's only one possible limitation.

For example, a 2 TB dataset could theoretically be processed by a machine with enough memory. But buying and maintaining a machine with enormous amounts of RAM may not be practical or economical.

The real question is:

When does processing data on a single machine stop being practical?

That's where distributed computing becomes interesting.

The 5 Vs of Big Data

A common way of describing Big Data is through the 5 Vs.

V	Meaning	Example
Volume	Amount of data	Terabytes of logs
Velocity	Speed at which data is generated	Millions of events per minute
Variety	Different types and formats	JSON, CSV, logs, images
Veracity	Data quality and reliability	Missing or inconsistent data
Value	Useful information extracted from data	Customer insights

Not every data problem involves all five.

But these dimensions help explain why traditional approaches can eventually become difficult to scale.

Why Can't We Just Use Pandas?

Let's start with something familiar.

Suppose we have a sales dataset:

import pandas as pd

df = pd.read_csv("sales.csv")

result = df.groupby("customer")["amount"].sum()

For a reasonably sized dataset, this is perfectly fine.

Pandas is an excellent tool for:

Data analysis
Data cleaning
Exploratory data analysis
Prototyping
Working with small and medium-sized datasets

But now imagine that the dataset is 500 GB.

Your laptop might have:

CPU: 8 cores
RAM: 16 GB

Trying to load hundreds of gigabytes into memory on that machine isn't practical.

And there's another important detail.

A CSV file's size on disk isn't necessarily the same as the memory required by the resulting DataFrame.

Pandas needs memory for:

Data values
Column structures
Indexes
Strings
Internal data structures
Other memory overhead

So a 10 GB CSV can require significantly more than 10 GB of RAM when loaded into memory.

Eventually, the single-machine approach reaches a limit.

The Single-Machine Problem

Imagine we have:

10 TB of data

but our server has:

64 GB RAM

We can't simply load the entire dataset into memory.

One option would be to buy a much larger machine.

That brings us to vertical scaling.

Vertical Scaling

Vertical scaling means making one machine more powerful.

For example:

Before

┌───────────────────┐
│      Server       │
│                   │
│   8 CPU cores     │
│   32 GB RAM       │
└───────────────────┘

          ↓

After

┌───────────────────┐
│      Server       │
│                   │
│   64 CPU cores    │
│   256 GB RAM      │
└───────────────────┘

This can work very well for many workloads.

But it has limitations.

Hardware becomes increasingly expensive, and eventually there are physical and economic limits to how large a single machine can become.

So we have another option.

Horizontal Scaling

Instead of buying one enormous machine, we can use multiple machines.

For example:

              10 TB Dataset
                    │
       ┌────────────┼────────────┐
       ↓            ↓            ↓
   Server 1     Server 2     Server 3
    3.3 TB        3.3 TB        3.4 TB

Each machine can process part of the dataset.

This is horizontal scaling.

Instead of:

One extremely powerful machine

we use:

Many machines working together

This leads us to an important concept named distributed computing.

What Is Distributed Computing?

Distributed computing is an approach where multiple computers work together over a network to solve a problem by dividing the workload among them.

For example:

                 Dataset
                    │
          ┌─────────┼─────────┐
          ↓         ↓         ↓
       Worker 1  Worker 2  Worker 3
          │         │         │
       Part A    Part B    Part C
          │         │         │
          └─────────┼─────────┘
                    ↓
             Combined Result

Instead of one machine processing everything, different machines process different portions of the workload.

And because they can work in parallel, the total processing time can potentially be reduced significantly.

But Distributed Computing Isn't Free

At first, distributed computing sounds simple:

"Just add more machines."

Unfortunately, it isn't that easy.

Imagine a cluster containing hundreds of machines.

Now we have new problems.

What if a machine fails?

Worker 1 ✓
Worker 2 ✓
Worker 3 ✗
Worker 4 ✓

What happens to the work assigned to Worker 3?

How do we divide the data?

We need to decide which machine processes which portion of the dataset.

How do machines communicate?

The machines need to exchange information over a network.

Network communication isn't free.

How do we combine the results?

Each machine produces partial results.

We eventually need to combine them into the final result.

What if one machine is much slower?

Suppose:

Worker 1 → 10 seconds
Worker 2 → 12 seconds
Worker 3 → 11 seconds
Worker 4 → 3 minutes

The entire operation may have to wait for the slow worker.

This is one of the challenges of distributed systems.

So We Have a New Problem

We've gone from:

Single Machine
      │
      ↓
Data becomes too large
      │
      ↓
Add more machines
      │
      ↓
Distributed Computing
      │
      ↓
Now we need to coordinate
      │
      ↓
Distributed Processing Frameworks

And this is where technologies such as Hadoop and later Apache Spark come into the picture.

Where Does Apache Spark Fit?

At a very high level:

             Large Dataset
                   │
                   ↓
        ┌─────────────────────┐
        │ Distributed         │
        │ Processing          │
        └──────────┬──────────┘
                   │
          ┌────────┴────────┐
          ↓                 ↓
       Hadoop             Spark

Hadoop introduced a widely used ecosystem for distributed storage and processing.

Spark later became popular as a powerful distributed computing engine capable of handling a wide variety of workloads.

We'll explore the history and differences in the next article.

An Important Clarification

It's easy to misunderstand this topic and conclude:

"Pandas is bad and Spark is good."

That's not the right way to think about it.

Pandas and Spark solve different problems.

For example:

Pandas
  ↓
Single-machine data analysis

while:

Spark
  ↓
Distributed data processing

And Spark isn't automatically better just because the dataset is large.

If you're processing a small dataset on your laptop, using Spark could introduce unnecessary complexity.

The right question is:

What processing architecture is appropriate for this workload?

That's an important mindset for a data engineer.

My First Hands-On Experiment

To understand this problem instead of simply reading about it, I created a CSV file containing 1 million rows and measured how long Pandas took to load it.

The experiment was simple:

Generate CSV
      ↓
Measure file size
      ↓
Check system RAM
      ↓
Load with Pandas
      ↓
Measure loading time

The code looked roughly like this:

import csv
import os
import time

import pandas as pd
import psutil

file_name = "millions_of_rows.csv"
total_rows = 1_000_000

start_time = time.perf_counter()

with open(file_name, "w", newline="") as file:
    writer = csv.writer(file)

    writer.writerow(["id", "name", "score"])

    for i in range(1, total_rows + 1):
        writer.writerow([
            i,
            f"Person_{i}",
            i % 100
        ])

creation_time = time.perf_counter() - start_time

print(f"CSV created in {creation_time:.2f} seconds")

file_size_bytes = os.path.getsize(file_name)
file_size_mb = file_size_bytes / (1024 ** 2)

ram = psutil.virtual_memory()

total_ram_gb = ram.total / (1024 ** 3)
available_ram_gb = ram.available / (1024 ** 3)

print(f"File size: {file_size_mb:.2f} MB")
print(f"Total RAM: {total_ram_gb:.2f} GB")
print(f"Available RAM: {available_ram_gb:.2f} GB")

print("Reading CSV using Pandas")

start_time = time.perf_counter()

df = pd.read_csv(file_name)

loading_time = time.perf_counter() - start_time

print(f"Rows loaded: {len(df):,}")
print(f"Columns loaded: {len(df.columns)}")
print(f"Loading time: {loading_time:.2f} seconds")

The purpose wasn't to prove that Pandas is slow.

The purpose was to observe what happens as the amount of data increases.

A useful follow-up experiment would be to compare:

100K rows
500K rows
1M rows
5M rows
10M rows

and record:

File size
Loading time
Memory consumption

This gives a much better intuition for the single-machine limitation.

The Bigger Picture

At this point, the evolution should look something like this:

                 Data
                  │
                  ↓
        Single-machine tools
        ┌─────────┴─────────┐
        │                   │
     Pandas             Databases
        │                   │
        └─────────┬─────────┘
                  │
                  ↓
           Data gets larger
                  │
                  ↓
       Single machine becomes
          less practical
                  │
                  ↓
       Horizontal Scaling
                  │
                  ↓
       Distributed Computing
                  │
                  ↓
            Hadoop / Spark

This is the foundation we need before learning Spark.

What I Learned

The biggest takeaway from this lesson for me was:

Spark isn't the starting point of the story. Distributed computing is.

Before learning Spark APIs, it's important to understand why distributing computation across multiple machines becomes necessary in the first place.

The key concepts I want to remember are:

1. Big Data

Data becomes challenging when its scale or processing requirements exceed what traditional approaches can handle efficiently.

2. Vertical Scaling

Make one machine more powerful.

3. Horizontal Scaling

Add more machines.

4. Distributed Computing

Multiple machines work together to solve a problem.

5. The Trade-off

Distributed systems provide scalability, but introduce complexity around:

Communication
Network transfer
Failure handling
Data partitioning
Coordination
Combining results

And this is exactly the problem that distributed processing frameworks are designed to help solve.

What's Next?

Now that we understand why distributed computing is necessary, we can look at the technology that popularized large-scale distributed data processing:

Apache Hadoop & MapReduce

In the next article, I'll explore:

What Hadoop is
Why Hadoop was created
What HDFS does
What MapReduce is
Map vs Reduce
The Shuffle phase
How a distributed job actually executes
Why MapReduce can become slow
And finally...

Why Apache Spark was created as a better approach for many workloads.

Follow the Series

This is Part 1 of my Apache Spark learning journey.

I'm learning Spark from the ground up and documenting what I learn along the way-not as an expert, but as a data engineer trying to understand distributed data processing properly.

If you're also learning Spark, feel free to follow along.

Next: Apache Spark for Beginners #2 - Hadoop, MapReduce, and the Problem Spark Solves

DEV Community: Sanjeev Kumar

Apache Spark for Beginners #1: Why Does Spark Exist?

What Is "Big Data"?

The 5 Vs of Big Data

Why Can't We Just Use Pandas?

The Single-Machine Problem

Vertical Scaling

Horizontal Scaling

What Is Distributed Computing?

But Distributed Computing Isn't Free

What if a machine fails?

How do we divide the data?

How do machines communicate?

How do we combine the results?

What if one machine is much slower?

So We Have a New Problem

Where Does Apache Spark Fit?

An Important Clarification

My First Hands-On Experiment

The Bigger Picture

What I Learned

1. Big Data

2. Vertical Scaling

3. Horizontal Scaling

4. Distributed Computing

5. The Trade-off

What's Next?

Apache Hadoop & MapReduce

Follow the Series