Your Spark Job Has 200 Executors. Why Is Only One of Them Working?

Understanding Data Skew in Apache Spark: The Hidden Reason Behind Slow Data Pipelines

Imagine you’re running a Spark job to process 2 TB of e-commerce data.

You have configured a cluster with 200 executors. Your code is correct. Your cluster has enough resources. Most tasks finish within a few minutes.

But one task keeps running.

Five minutes pass.

Ten minutes pass.

Thirty minutes later, that single task is still running while most of your cluster sits idle.

You increase the number of executors from 200 to 300.

Nothing changes.

You increase the memory.

Still nothing.

Eventually, your pipeline finishes, but it takes hours longer than expected.

What went wrong?

One possible reason is Data Skew.

And if you work with Apache Spark, understanding this concept can fundamentally change how you debug slow pipelines.

1. First, Understand How Spark Distributes Work

Apache Spark processes data in parallel by dividing it into partitions.

Think of a company that needs to process one million customer orders.

Instead of asking one employee to process every order, the company distributes the work across 100 employees.

If every employee receives approximately 10,000 orders, the workload is balanced.

But imagine one employee receives 700,000 orders while the remaining 99 employees share just 300,000.

Even if 99 employees finish early, the company cannot finish until the overloaded employee completes their work.

Distributed computing works similarly.

Spark distributes partitions across tasks, and tasks execute on available executor resources.

Parallelism helps only when the work is distributed reasonably well.

Having more executors does not automatically mean your workload is balanced.

1. What Exactly Is Data Skew?

Data skew occurs when data is distributed unevenly across partitions, causing some tasks to process significantly more data than others.

Let’s take a simple example.

Suppose an e-commerce company has 100 million order records.

The company wants to calculate total sales for each seller.

A simplified dataset looks like this:

Seller ID Number of orders
Seller A 70 million
Seller B 10 million
Seller C 8 million
Other sellers 12 million combined

Seller A alone accounts for 70% of all orders.

If Spark performs a shuffle-based aggregation by seller_id, all records for a given seller generally need to be brought together for that aggregation stage.

The partition responsible for Seller A may receive a much larger workload than partitions handling smaller sellers.

The result?

One task processes tens of millions of records while other tasks finish quickly.

That is the practical impact of data skew.

Important distinction: A frequent value does not automatically cause a performance problem in every Spark operation. The impact depends on the operation, partitioning strategy, data volume, and execution plan.

1. Why Adding More Executors Doesn’t Necessarily Help

This is where many engineers make an expensive mistake.

They see a slow Spark job and immediately increase cluster resources.

But consider what happens when Spark shuffles data by a key.

If 70 million records belong to the same seller, simply increasing the number of shuffle partitions does not necessarily distribute that seller’s records across those partitions.

For a standard aggregation or join, records with the same key may still need to be processed together at a particular stage.

Adding executors can increase available computing capacity, but it cannot automatically eliminate an algorithmic bottleneck.

Before scaling your cluster, ask:

• Are a few keys responsible for most of the records?
• Are some tasks processing much more data than others?
• Is the bottleneck caused by skew, insufficient memory, disk spilling, or something else?
• Which operation introduces the expensive shuffle?

The Spark UI is often the best place to begin this investigation.

1. How Do You Detect Data Skew in Production?

When debugging a slow pipeline, I would investigate the following signals.

A. Task duration

Compare task execution times within the same stage.

If most tasks finish in 20 seconds but a few take 15 minutes, investigate why.

Uneven duration alone does not prove skew, but it is a useful signal.

B. Shuffle read size

Inspect how much shuffled data each task reads.

If a small number of tasks read substantially more data than the rest, skew may be contributing to the bottleneck.

C. Memory and disk spilling

Large partitions may create substantial memory pressure. Spark can spill intermediate data to disk, increasing execution time.

D. The data itself

Run a frequency analysis on important join and aggregation keys.

For example:

SELECT
seller_id,
COUNT(*) AS order_count
FROM orders
GROUP BY seller_id
ORDER BY order_count DESC
LIMIT 20;

This query helps identify unusually frequent seller IDs.

However, key frequency is only one part of the diagnosis. You should also inspect the physical execution plan and Spark UI to understand how the workload is actually distributed.

1. Four Ways to Handle Data Skew

There is no single solution that works for every workload. The correct approach depends on whether the skew occurs during a join, aggregation, or another shuffle-heavy operation.

Technique 1: Broadcast Join

Suppose you have:

• An orders table containing 500 million rows
• A seller dimension table containing only 20,000 rows

If the smaller table fits safely within the available memory, a broadcast join can distribute it to the executors processing the larger dataset.

Example:

from pyspark.sql.functions import broadcast
result = orders.join(
broadcast(sellers),
on="seller_id",
how="left"
)

This can avoid shuffling the large orders dataset for the join.

But there is an important limitation: broadcasting a table that is too large can create memory pressure or even cause executor failures. Always validate the table’s size and the cluster’s available resources.

Broadcast joins are especially useful when the join’s smaller side is genuinely small enough to broadcast.

Technique 2: Adaptive Query Execution (AQE)

Modern Spark versions support Adaptive Query Execution, which can adjust parts of a physical query plan using runtime statistics.

For skewed sort-merge joins, Spark can identify certain skewed partitions and split them into smaller pieces, processing them alongside replicated pieces of the other join side.

A common configuration is:

spark.conf.set(
"spark.sql.adaptive.enabled",
"true"
)
spark.conf.set(
"spark.sql.adaptive.skewJoin.enabled",
"true"
)

These settings enable AQE and its skew-join optimization.

The precise behavior depends on the Spark version, thresholds, and query plan. AQE is not a universal fix for every skewed aggregation or every slow task.

Technique 3: Salting

Salting is a manual technique that can help when a few keys dominate the workload.

Suppose Seller A has 70 million orders.

Instead of processing all of Seller A’s records under one key, we can create multiple temporary keys:

• Seller A + Salt 0
• Seller A + Salt 1
• Seller A + Salt 2
• Seller A + Salt 3
• Seller A + Salt 4

This creates more opportunities for parallel processing.

For aggregations such as SUM, we can calculate partial totals for each salted key and then combine those totals to obtain the final result.

For joins, we can salt the skewed side and replicate the corresponding rows from the smaller side across the relevant salt values.

But salting introduces additional work and complexity. It must be designed carefully to preserve the correctness of the result.

Technique 4: Revisit the Data Model

Sometimes the best optimization is not a Spark configuration change.

Ask why one key owns such a large share of the workload.

Can the business process generate a more balanced key? Can the aggregation be split into partial and final stages? Can the data be compacted or filtered earlier? Is the join actually necessary at its current grain?

The best solution may come from changing the computation rather than simply adding hardware.

1. A Common Interview Trap: Does Repartitioning Always Fix Skew?

No.

Consider:

df.repartition(200, "seller_id")

This repartitions the data using seller_id, but it does not guarantee an even distribution of records across the resulting partitions.

If Seller A dominates the dataset, its records can still be concentrated in one partition.

Increasing the number of partitions does not guarantee that a highly frequent key will be split across them.

This is why engineers must distinguish between two concepts:

Partitioning: How data is organized and distributed across partitions.

Skew: How unevenly the workload or data is distributed across those partitions.

They are related, but they are not the same thing.

1. What I Would Do During a Production Incident

If a critical Spark pipeline suddenly becomes much slower, I would follow a structured investigation:

1. Identify the slow stage in the Spark UI.
2. Compare task durations, shuffle reads, and spill metrics.
3. Check whether a few partitions are disproportionately large.
4. Inspect join keys and aggregation keys for highly frequent values.
5. Review the physical execution plan and whether AQE is active.
6. Test the smallest appropriate optimization, such as a broadcast join, skew-join handling, or a two-stage aggregation.
7. Compare the result’s correctness and execution metrics before and after the change.

I would not immediately increase the executor count without first identifying the bottleneck.

A larger cluster can help with insufficient overall capacity, but it may provide little improvement when one skewed task determines the completion time.

1. The Bigger Lesson for Data Engineers

Data Engineering is not just about writing transformations that return the correct output.

It is about understanding how data moves, where computation happens, how resources are consumed, and why a pipeline behaves differently at production scale.

Data skew connects several essential skills:

• Distributed computing
• SQL joins and aggregations
• Spark execution plans
• Shuffle mechanics
• Memory and disk management
• Data modeling
• Production debugging

When you understand these relationships, Spark optimization becomes a process of diagnosing evidence rather than randomly changing configurations.

Final Thoughts

If your Spark job is slow, don’t ask only:

How many executors do I need?

Ask a more useful question:

Is my data distributed in a way that allows those executors to work efficiently?

That single question can lead you toward the real bottleneck.

At PrepNPlaced, I focus on helping aspiring and working Data Engineers understand concepts through practical scenarios, technical depth, and production-oriented problem-solving.

I’m Durgesh Yadav, and I believe the goal is not simply to learn tools like Spark, Databricks, or Hadoop. It is to understand why they behave the way they do—and how to make them reliable at scale.

If you’re learning Data Engineering, save this article and revisit the examples when you next troubleshoot a slow Spark job.

More learning resources: https://www.prepnplaced.com/