Why Is Spark Slow??

hirayuki — Sun, 01 Dec 2024 14:40:49 +0000

Why Is Spark Slow??

Starting with an eye-catching title, "Why is Spark slow??," it's important to note that calling Spark "slow" can mean various things. Is it slow at aggregations? Data loading? Different cases exist. Also, "Spark" is a broad term, and its performance depends on factors like the programming language and usage context. So, let's refine the title to be more precise before diving in.

Since I primarily use Spark with Python on Databricks, I'll narrow the scope further.

The refined title will be:

"First Impressions of Spark: 'I Heard It Was Fast, But Why Does It Feel Slow?' A Beginner's Perspective"

Motivation for Writing (Casual Thoughts)

As someone who works extensively with pandas, NumPy, and machine learning libraries, I admired the allure of Spark's ability to handle big data with parallel and distributed processing. When I finally got to use Spark for work, I was puzzled by scenarios where it seemed slower than pandas. Unsure of what was wrong, I discovered several insights and would like to share them.

When Does Your Spark Become Slow?

Before Getting to the Main Topic

Let's briefly cover Spark's basic architecture.

(Cluster Mode Overview)

A Spark cluster consists of Worker Nodes, which perform the actual processing, and a Driver Node, which coordinates and plans the execution. This architecture influences everything discussed below, so keep it in mind.

Now, onto the main points.

1. The Dataset Isn’t Large Enough

Spark is optimized for large-scale data processing, though it can handle small datasets as well. However, take a look at this benchmark:

(Benchmarking Apache Spark on a Single Node Machine)

The results show that for datasets under 15GB, pandas outperforms Spark in aggregation tasks. Why? In a nutshell, the overhead of Spark's optimizations outweighs the benefits for small datasets.

The link shows cases where Spark isn't slower, but these are often in a local cluster mode. For standalone setups, smaller datasets can be a disadvantage due to network communication overhead between nodes.

pandas: Processes everything in-memory on a single machine, with no network or storage I/O.
Spark: Uses RDDs (Resilient Distributed Datasets), involves network communication between Workers (if distributed), and incurs overhead in organizing data for parallel processing.

2. Understanding Lazy Evaluation

Spark employs lazy evaluation, meaning transformations are not executed immediately but deferred until an action (e.g., collect, count, show) triggers computation.

Example (pandas):

df = spark.read.table("tpch.lineitem").limit(1000).toPandas()
df["l_tax_percentage"] = df["l_tax"] * 100
for l_orderkey, group_df in df.groupby("l_orderkey"):
    print(l_orderkey, group_df["l_tax_percentage"].mean())

Execution time: 3.04 seconds

Equivalent in Spark:

from pyspark.sql import functions as F
sdf = spark.read.table("tpch.lineitem").limit(1000)
sdf = sdf.withColumn("l_tax_percentage", F.col("l_tax") * 100)

for row in sdf.select("l_orderkey").distinct().collect():
    grouped_sdf = sdf.filter(F.col("l_orderkey") == row.l_orderkey).groupBy("l_orderkey").agg(
        F.mean("l_tax_percentage").alias("avg_l_tax_percentage")
    )
    print(grouped_sdf.show())

Execution time: Still running after 3 minutes.

Why?

Lazy Evaluation: All transformations are queued and only executed during an action like show.
Driver-to-Worker Communication: Operations like collect and show involve data transfer from Workers to the Driver, causing delays.

The Spark code effectively does this in pandas:

for l_orderkey, group_df in df.groupby("l_orderkey"):
    df["l_tax_percentage"] = df["l_tax"] * 100
    print(l_orderkey, group_df["l_tax_percentage"].mean())

Avoid such patterns by using Spark's cache or restructuring the logic to minimize repeated calculations.

3. Watch Out for Shuffles

https://spark.apache.org/docs/latest/rdd-programming-guide.html#shuffle-operations

Shuffles occur when data is redistributed across Workers, typically during operations like groupByKey, join, or repartition. Shuffles can be slow due to:

Network Communication between nodes.
Global Sorting and Aggregation of data across partitions.

For example, having more Workers doesn't always improve performance during a shuffle.

32GB x 8 Workers can be slower than 64GB x 4 Workers, as fewer Workers reduce inter-node communication.

Conclusion

Did you find this helpful? Spark is an excellent tool when used effectively. Beyond speeding up large-scale data processing, Spark shines with its scalable resource management, especially in the cloud.

Try Spark to optimize your data operations and management!

DEV Community: hirayuki

Why Is Spark Slow??

Why Is Spark Slow??

Motivation for Writing (Casual Thoughts)

When Does Your Spark Become Slow?

Before Getting to the Main Topic

1. The Dataset Isn’t Large Enough

2. Understanding Lazy Evaluation

3. Watch Out for Shuffles

Conclusion