In my previous article, I explored why Apache Spark exists, what Big Data means, and why processing large datasets on a single machine can become impractical.
I learned that one way to handle growing data is to distribute the workload across multiple machines.
But this raises some important questions:
- How do we store data across multiple machines?
- How do we process that distributed data?
- How do we handle machine failures?
- How do we combine the results produced by different machines?
To understand how Apache Spark fits into this picture, I first need to understand the technologies that helped popularize large-scale distributed data processing.
In this article, I'll explore Apache Hadoop, HDFS, and MapReduce.
1. The problem: We have more data than one machine can handle
Imagine a company collects 10 TB of sales data.
The company wants to answer a simple business question:
What is the total sales amount for each product?
The dataset might look like this:
product,amount
Laptop,1000
Phone,500
Laptop,2000
Tablet,700
Phone,300
For a small dataset, Python and Pandas can handle this easily.
import pandas as pd
df = pd.read_csv("sales.csv")
result = df.groupby("product")["amount"].sum()
print(result)
But what if the dataset contains 10 TB of records?
A single machine may not have enough memory, processing capacity, or storage to handle the workload efficiently.
One option is to use multiple machines.
However, distributing data across machines introduces new challenges. We need a way to store the data, coordinate processing, handle failures, and combine results.
This is where Hadoop becomes relevant.
2. What is Apache Hadoop?
Apache Hadoop is an open-source framework and ecosystem designed to support distributed storage and processing of large datasets across clusters of machines.
At a high level, two important Hadoop components are:
- HDFS: Hadoop Distributed File System, which provides distributed file storage.
- MapReduce: A programming model and processing framework for distributed batch computation.
They solve different problems.
Apache Hadoop
|
+--------+--------+
| |
v v
HDFS MapReduce
| |
Store data Process data
across nodes across nodes
Think of it this way:
- HDFS answers: Where is the data stored?
- MapReduce answers: How can we process the data across multiple machines?
These are foundational concepts, but Hadoop is broader than these two components.
3. Understanding HDFS
HDFS stands for Hadoop Distributed File System.
It is designed to store large files across multiple machines in a cluster.
Why do we need distributed storage?
Imagine a 10 TB file.
Instead of storing the entire file on one machine, HDFS divides files into blocks and distributes those blocks across multiple machines.
For example, imagine a simplified cluster:
sales.csv
|
10 TB of data
|
+---------+---------+
| | |
v v v
Node 1 Node 2 Node 3
Block A Block B Block C
This diagram is conceptual. In a real HDFS cluster, a file is divided into many blocks, which are distributed across DataNodes.
What are NameNode and DataNode?
HDFS has two important roles.
NameNode
The NameNode manages filesystem metadata, including information about files, directories, and which DataNodes hold the file blocks.
It generally does not store the actual file contents.
DataNode
DataNodes store the actual data blocks and serve read and write requests.
A simplified view looks like this:
Client
|
v
NameNode
(Metadata management)
|
Knows block locations
|
+---------+---------+
| | |
v v v
DataNode DataNode DataNode
Block A Block B Block C
The client consults the NameNode for metadata and then accesses the appropriate DataNodes to read or write the data.
What happens if a machine fails?
Suppose one DataNode fails.
If a file block has another available replica on a different DataNode, the data can still be accessed.
HDFS uses block replication to improve fault tolerance. The replication factor is configurable; a factor of three is a common default in traditional HDFS deployments.
This is one of the advantages of distributed storage.
Does every machine already have storage?
Yes. Every machine can have its own local disk.
But local storage and HDFS are not the same thing.
A file stored on Node 1's local disk is not automatically accessible through Node 2's local filesystem.
HDFS provides a coordinated filesystem abstraction across the cluster, along with features such as block management and replication.
Important: HDFS is not mandatory for every Spark application. Spark can read data from local files, cloud object storage such as Amazon S3, and many other supported data sources.
HDFS is important for understanding Hadoop's architecture, not because every Spark job must use it.
4. What is MapReduce?
Now that we understand distributed storage, let's look at distributed processing.
Suppose our sales data is distributed across several machines.
We want to calculate the total sales amount for each product.
MapReduce is a programming model used to process data in parallel across a cluster.
Its core computation involves two main functions:
- Map
- Reduce
A distributed MapReduce job also involves shuffling and sorting intermediate data between these stages.
Let's understand the process using our sales example.
5. Step 1: Map
The Map phase processes input records and produces intermediate key-value pairs.
Suppose two machines process different portions of the sales data.
Machine 1 receives:
Laptop,1000
Phone,500
Laptop,2000
The Map function produces:
(Laptop, 1000)
(Phone, 500)
(Laptop, 2000)
Machine 2 receives:
Tablet,700
Phone,300
It produces:
(Tablet, 700)
(Phone, 300)
Each mapper processes its input split independently.
At this point, we have intermediate results, but we haven't calculated the total sales for each product.
For example, the two Phone records still need to be brought together.
6. Step 2: Shuffle and Sort
This is one of the most important concepts in distributed data processing.
After the Map phase, intermediate records are partitioned by key, transferred to the appropriate reducers, and grouped by key.
This process is commonly called Shuffle and Sort.
Before the shuffle, the intermediate results might look like this:
Mapper 1 Mapper 2
| |
Laptop -> 1000 Tablet -> 700
Phone -> 500 Phone -> 300
Laptop -> 2000
The framework redistributes records so that values belonging to the same key reach the same reducer.
Conceptually, the grouped data becomes:
Laptop -> [1000, 2000]
Phone -> [500, 300]
Tablet -> [700]
Why is this necessary?
Because the system needs all the values for a given product to be available to the reducer that calculates that product's total.
Shuffle involves network communication and data movement. Depending on the workload, it can be expensive.
Remember this concept. We'll encounter shuffle again when studying Apache Spark, where it is essential to understanding performance.
7. Step 3: Reduce
The Reduce phase processes each key and its associated values.
For example:
Laptop -> [1000, 2000]
The reducer adds the values:
Laptop -> 3000
Similarly:
Phone -> [500, 300]
Tablet -> [700]
The final result becomes:
product,total_sales
Laptop,3000
Phone,800
Tablet,700
We have now calculated the total sales for every product using distributed processing.
One important detail: MapReduce is a general programming model. The Map and Reduce functions depend on the problem being solved; summing sales is just one example.
8. Putting the complete workflow together
Let's connect HDFS and MapReduce.
Sales Data
|
v
HDFS
(Distributed storage)
|
v
Input Splits
|
v
MAP
Process input records
|
v
Intermediate key-values
|
v
SHUFFLE & SORT
Group records by key
|
v
REDUCE
Calculate sales totals
|
v
Final Results
The storage and computation layers have different responsibilities.
HDFS stores the data. MapReduce processes it.
In a traditional Hadoop deployment, the framework handles much of the distributed execution, task scheduling, and recovery from failures.
This saves developers from having to implement every aspect of cluster coordination themselves.
9. Why was Hadoop important?
Hadoop helped make large-scale distributed data processing accessible using clusters of machines.
Instead of depending exclusively on one extremely powerful server, organizations could distribute data and processing across multiple nodes.
Its design also accounted for the reality that machines can fail.
However, distributed computing does not automatically make every workload faster. Network communication, coordination, disk I/O, and data skew can all affect performance.
This leads to another question.
If MapReduce can process huge datasets, why was Apache Spark developed?
10. What limitations motivated Spark?
MapReduce is effective for many batch-processing workloads, but complex data pipelines can require multiple MapReduce jobs.
Imagine this pipeline:
Read Data
|
v
Filter
|
v
Join
|
v
Aggregate
|
v
Sort
|
v
Write Results
Depending on how the pipeline is implemented, intermediate results may need to be materialized to disk between jobs.
That can introduce additional disk I/O, scheduling overhead, and latency.
This is particularly relevant when a workload repeatedly reuses intermediate data, such as iterative machine learning algorithms.
Apache Spark was developed to provide a more general-purpose distributed computation engine, with support for caching and reusing intermediate data in memory when appropriate.
Spark can also spill data to disk when needed, so it is not an in-memory-only system.
The distinction is not simply that MapReduce is slow and Spark is fast. Actual performance depends on the workload, execution plan, storage, cluster configuration, and other factors.
Spark provides a more flexible execution model for many kinds of data-processing workloads.
11. Hadoop vs. Spark: A high-level comparison
| Aspect | Hadoop MapReduce | Apache Spark |
|---|---|---|
| Primary role | Distributed batch processing | General-purpose distributed computation |
| Execution model | Map and Reduce stages | Directed acyclic graph (DAG) of operations |
| Intermediate data | Often materialized between jobs | Can be reused in memory when appropriate |
| Iterative workloads | Often require repeated jobs and materialization | Caching can make repeated computation more efficient |
| Storage | Commonly paired with HDFS | Supports HDFS, cloud storage, local files, and other sources |
| Processing style | Primarily batch-oriented | Batch, streaming, SQL, and other workloads |
This is a simplified comparison. Modern Hadoop ecosystems contain more than MapReduce, and Spark can use disk as well as memory.
The main takeaway is that Spark offers a flexible execution engine that can optimize multi-stage data-processing workloads.
12. Key takeaways
Here's what I learned from exploring Hadoop and MapReduce:
- Hadoop provides an ecosystem for distributed data storage and processing.
- HDFS stores files as blocks distributed across machines and supports replication.
- The NameNode manages filesystem metadata, while DataNodes store data blocks.
- Map transforms input records into intermediate key-value pairs.
- Shuffle and Sort redistributes and groups intermediate records by key.
- Reduce processes the grouped values to produce results.
- Distributed processing introduces challenges such as network communication, coordination, and failure handling.
- Apache Spark provides a flexible distributed execution engine and can reuse intermediate data in memory when appropriate.
- Spark does not require HDFS; it supports multiple storage systems.
What's next?
In the next article, I'll explore Apache Spark itself.
I'll cover:
- What Spark is and what problems it solves.
- How Spark differs from Hadoop MapReduce.
- What PySpark is.
- How to install PySpark locally.
- How to run a first Spark application.
The goal is to move from understanding the motivation behind distributed processing to actually running code.
This is Part 2 of my Apache Spark for Beginners learning series. I'm documenting the journey as I build my understanding of distributed data processing, one concept at a time.
Top comments (0)