DEV Community: Durgesh Yadav

Your Spark Job Has 200 Executors. Why Is Only One of Them Working?

Durgesh Yadav — Thu, 15 Oct 2026 22:31:31 +0000

Understanding Data Skew in Apache Spark: The Hidden Reason Behind Slow Data Pipelines

Imagine you’re running a Spark job to process 2 TB of e-commerce data.

You have configured a cluster with 200 executors. Your code is correct. Your cluster has enough resources. Most tasks finish within a few minutes.

But one task keeps running.

Five minutes pass.

Ten minutes pass.

Thirty minutes later, that single task is still running while most of your cluster sits idle.

You increase the number of executors from 200 to 300.

Nothing changes.

You increase the memory.

Still nothing.

Eventually, your pipeline finishes, but it takes hours longer than expected.

What went wrong?

One possible reason is Data Skew.

And if you work with Apache Spark, understanding this concept can fundamentally change how you debug slow pipelines.

First, Understand How Spark Distributes Work

Apache Spark processes data in parallel by dividing it into partitions.

Think of a company that needs to process one million customer orders.

Instead of asking one employee to process every order, the company distributes the work across 100 employees.

If every employee receives approximately 10,000 orders, the workload is balanced.

But imagine one employee receives 700,000 orders while the remaining 99 employees share just 300,000.

Even if 99 employees finish early, the company cannot finish until the overloaded employee completes their work.

Distributed computing works similarly.

Spark distributes partitions across tasks, and tasks execute on available executor resources.

Parallelism helps only when the work is distributed reasonably well.

Having more executors does not automatically mean your workload is balanced.

What Exactly Is Data Skew?

Data skew occurs when data is distributed unevenly across partitions, causing some tasks to process significantly more data than others.

Let’s take a simple example.

Suppose an e-commerce company has 100 million order records.

The company wants to calculate total sales for each seller.

A simplified dataset looks like this:

Seller ID Number of orders
Seller A 70 million
Seller B 10 million
Seller C 8 million
Other sellers 12 million combined

Seller A alone accounts for 70% of all orders.

If Spark performs a shuffle-based aggregation by seller_id, all records for a given seller generally need to be brought together for that aggregation stage.

The partition responsible for Seller A may receive a much larger workload than partitions handling smaller sellers.

The result?

One task processes tens of millions of records while other tasks finish quickly.

That is the practical impact of data skew.

Important distinction: A frequent value does not automatically cause a performance problem in every Spark operation. The impact depends on the operation, partitioning strategy, data volume, and execution plan.

Why Adding More Executors Doesn’t Necessarily Help

This is where many engineers make an expensive mistake.

They see a slow Spark job and immediately increase cluster resources.

But consider what happens when Spark shuffles data by a key.

If 70 million records belong to the same seller, simply increasing the number of shuffle partitions does not necessarily distribute that seller’s records across those partitions.

For a standard aggregation or join, records with the same key may still need to be processed together at a particular stage.

Adding executors can increase available computing capacity, but it cannot automatically eliminate an algorithmic bottleneck.

Before scaling your cluster, ask:

Are a few keys responsible for most of the records?
Are some tasks processing much more data than others?
Is the bottleneck caused by skew, insufficient memory, disk spilling, or something else?
Which operation introduces the expensive shuffle?

The Spark UI is often the best place to begin this investigation.

How Do You Detect Data Skew in Production?

When debugging a slow pipeline, I would investigate the following signals.

A. Task duration

Compare task execution times within the same stage.

If most tasks finish in 20 seconds but a few take 15 minutes, investigate why.

Uneven duration alone does not prove skew, but it is a useful signal.

B. Shuffle read size

Inspect how much shuffled data each task reads.

If a small number of tasks read substantially more data than the rest, skew may be contributing to the bottleneck.

C. Memory and disk spilling

Large partitions may create substantial memory pressure. Spark can spill intermediate data to disk, increasing execution time.

D. The data itself

Run a frequency analysis on important join and aggregation keys.

For example:

SELECT
seller_id,
COUNT(*) AS order_count
FROM orders
GROUP BY seller_id
ORDER BY order_count DESC
LIMIT 20;

This query helps identify unusually frequent seller IDs.

However, key frequency is only one part of the diagnosis. You should also inspect the physical execution plan and Spark UI to understand how the workload is actually distributed.

Four Ways to Handle Data Skew

There is no single solution that works for every workload. The correct approach depends on whether the skew occurs during a join, aggregation, or another shuffle-heavy operation.

Technique 1: Broadcast Join

Suppose you have:

An orders table containing 500 million rows
A seller dimension table containing only 20,000 rows

If the smaller table fits safely within the available memory, a broadcast join can distribute it to the executors processing the larger dataset.

Example:

from pyspark.sql.functions import broadcast
result = orders.join(
broadcast(sellers),
on="seller_id",
how="left"
)

This can avoid shuffling the large orders dataset for the join.

But there is an important limitation: broadcasting a table that is too large can create memory pressure or even cause executor failures. Always validate the table’s size and the cluster’s available resources.

Broadcast joins are especially useful when the join’s smaller side is genuinely small enough to broadcast.

Technique 2: Adaptive Query Execution (AQE)

Modern Spark versions support Adaptive Query Execution, which can adjust parts of a physical query plan using runtime statistics.

For skewed sort-merge joins, Spark can identify certain skewed partitions and split them into smaller pieces, processing them alongside replicated pieces of the other join side.

A common configuration is:

spark.conf.set(
"spark.sql.adaptive.enabled",
"true"
)
spark.conf.set(
"spark.sql.adaptive.skewJoin.enabled",
"true"
)

These settings enable AQE and its skew-join optimization.

The precise behavior depends on the Spark version, thresholds, and query plan. AQE is not a universal fix for every skewed aggregation or every slow task.

Technique 3: Salting

Salting is a manual technique that can help when a few keys dominate the workload.

Suppose Seller A has 70 million orders.

Instead of processing all of Seller A’s records under one key, we can create multiple temporary keys:

Seller A + Salt 0
Seller A + Salt 1
Seller A + Salt 2
Seller A + Salt 3
Seller A + Salt 4

This creates more opportunities for parallel processing.

For aggregations such as SUM, we can calculate partial totals for each salted key and then combine those totals to obtain the final result.

For joins, we can salt the skewed side and replicate the corresponding rows from the smaller side across the relevant salt values.

But salting introduces additional work and complexity. It must be designed carefully to preserve the correctness of the result.

Technique 4: Revisit the Data Model

Sometimes the best optimization is not a Spark configuration change.

Ask why one key owns such a large share of the workload.

Can the business process generate a more balanced key? Can the aggregation be split into partial and final stages? Can the data be compacted or filtered earlier? Is the join actually necessary at its current grain?

The best solution may come from changing the computation rather than simply adding hardware.

A Common Interview Trap: Does Repartitioning Always Fix Skew?

No.

Consider:

df.repartition(200, "seller_id")

This repartitions the data using seller_id, but it does not guarantee an even distribution of records across the resulting partitions.

If Seller A dominates the dataset, its records can still be concentrated in one partition.

Increasing the number of partitions does not guarantee that a highly frequent key will be split across them.

This is why engineers must distinguish between two concepts:

Partitioning: How data is organized and distributed across partitions.

Skew: How unevenly the workload or data is distributed across those partitions.

They are related, but they are not the same thing.

What I Would Do During a Production Incident

If a critical Spark pipeline suddenly becomes much slower, I would follow a structured investigation:

Identify the slow stage in the Spark UI.
Compare task durations, shuffle reads, and spill metrics.
Check whether a few partitions are disproportionately large.
Inspect join keys and aggregation keys for highly frequent values.
Review the physical execution plan and whether AQE is active.
Test the smallest appropriate optimization, such as a broadcast join, skew-join handling, or a two-stage aggregation.
Compare the result’s correctness and execution metrics before and after the change.

I would not immediately increase the executor count without first identifying the bottleneck.

A larger cluster can help with insufficient overall capacity, but it may provide little improvement when one skewed task determines the completion time.

The Bigger Lesson for Data Engineers

Data Engineering is not just about writing transformations that return the correct output.

It is about understanding how data moves, where computation happens, how resources are consumed, and why a pipeline behaves differently at production scale.

Data skew connects several essential skills:

Distributed computing
SQL joins and aggregations
Spark execution plans
Shuffle mechanics
Memory and disk management
Data modeling
Production debugging

When you understand these relationships, Spark optimization becomes a process of diagnosing evidence rather than randomly changing configurations.

Final Thoughts

If your Spark job is slow, don’t ask only:

How many executors do I need?

Ask a more useful question:

Is my data distributed in a way that allows those executors to work efficiently?

That single question can lead you toward the real bottleneck.

At PrepNPlaced, I focus on helping aspiring and working Data Engineers understand concepts through practical scenarios, technical depth, and production-oriented problem-solving.

I’m Durgesh Yadav, and I believe the goal is not simply to learn tools like Spark, Databricks, or Hadoop. It is to understand why they behave the way they do—and how to make them reliable at scale.

If you’re learning Data Engineering, save this article and revisit the examples when you next troubleshoot a slow Spark job.

More learning resources: https://www.prepnplaced.com/

this is something mostly everyone goes through!

Durgesh Yadav — Fri, 25 Sep 2026 07:12:42 +0000

Durgesh Yadav

Sep 25

5 SQL patterns that run fine and still return the wrong answer

#career #dataengineering #sql #beginners

4 min read

5 SQL patterns that run fine and still return the wrong answer

Durgesh Yadav — Fri, 25 Sep 2026 05:40:42 +0000

A database table is really just a spreadsheet. Rows are records — one row
per customer, one row per order. Columns are the fields — a name, a date,
an amount. SQL is the language you use to ask that spreadsheet questions:
show me these rows, combine these two sheets, add these up.

The five things below aren't about learning more SQL words. They're about
five specific moments where a question you ask gets answered technically
correctly, but not in the way you meant — and nothing warns you. No error
message. Just a wrong number that looks right.

1. Asking for "everyone" and getting "only some people"

Say you have a table of customers and a table of orders. You want a list
of every customer, showing their order if they have one, and nothing if
they don't — because you still want to see the customers with no orders.

sql
SELECT c.customer_id, o.order_id
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
WHERE o.order_date >= '2026-01-01';

The LEFT JOIN part does say "keep everyone from the customers list."
But the WHERE line underneath quietly overrides it. Here's why: a
customer with no order gets a blank (called NULL) where their order date
would be. And "is this blank date after 1 January" doesn't have a yes or
no answer — it's neither true nor false, it's just undefined. SQL treats
undefined the same as no, so that customer gets thrown out.

You asked for everyone. You got only the people who also happen to have a
recent order. Nothing crashed. Nothing warned you. The list is just
quietly shorter than you think it is.

The picture makes this easier to see than the sentence does:

The fix is to move the date condition up into the matching step instead
of the filtering step:

sql
LEFT JOIN orders o
ON c.customer_id = o.customer_id
AND o.order_date >= '2026-01-01'

Now the date check only decides which order to attach, not whether the
customer gets kept.

2. A "give me everyone except these people" list that comes back empty

sql
SELECT * FROM customers
WHERE customer_id NOT IN (SELECT customer_id FROM cancelled_orders);

Plain English: show me every customer who isn't in the cancelled-orders
list. Reasonable ask.

Here's the trap. If even one row in cancelled_orders has a blank
customer_id — maybe a cancellation that was logged before anyone was
assigned to it — the whole comparison breaks. SQL can't say "is this
customer definitely not equal to a blank," so it refuses to say yes for
anyone at all. The entire result becomes empty, for a reason that has
nothing to do with your actual customers.

Swap it for asking the question a different way — not "is my ID absent
from that list," but "does a cancelled order exist for me":

sql
SELECT * FROM customers c
WHERE NOT EXISTS (
SELECT 1 FROM cancelled_orders x
WHERE x.customer_id = c.customer_id
);

This version isn't confused by a blank value sitting somewhere else in
the list. If you don't know for certain that a column can never contain a
blank, treat this as the safer way to ask the question.

3. Two ways of counting that quietly disagree

sql
SELECT COUNT(*) AS all_rows, COUNT(phone_number) AS with_phone
FROM customers;

Counting sounds like it should only have one right answer. It doesn't.
COUNT(*) counts every row, full stop. COUNT(phone_number) counts only
the rows where that specific field actually has something in it —
blanks don't get counted.

If those two numbers come back different, that gap is telling you exactly
how many customers have no phone number on file. It's a genuinely useful
one-line check for "how messy is this data" before you build anything
more complicated on top of it.

4. Adding things up in order, and getting the order wrong

sql
SELECT
order_date,
SUM(daily_revenue) OVER (ORDER BY order_date) AS running_total
FROM daily_sales;

This is meant to build a running total — day one's number, then day one
plus day two, then day one plus two plus three, and so on. Most of the
time it does exactly that.

The exception: if two rows share the exact same date, some databases
default to treating tied dates as one combined step rather than adding
them one at a time. Your running total can jump differently than expected
right at the point where a date repeats, and it's easy to miss because
the rest of the total looks completely normal.

Being explicit fixes it:

sql
SUM(daily_revenue) OVER (
ORDER BY order_date
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
)

This spells out "add strictly one row at a time," which removes the
ambiguity about what to do with ties.

5. A family tree that's missing the person at the top

sql
SELECT e.employee_name, m.employee_name AS manager_name
FROM employees e
JOIN employees m ON e.manager_id = m.employee_id;

This is a table joined to itself, to line up each employee with their
manager's name. It works for almost everyone — except whoever has no
manager at all, like the CEO or the founder. That one row has a blank
where the manager should be, and a plain join treats a blank as "doesn't
match," so that person disappears from the results entirely.

You'll get a org chart that looks complete and is quietly missing exactly
one row — the one at the very top.

sql
LEFT JOIN employees m ON e.manager_id = m.employee_id

Same lesson as the very first pattern: whenever a match can land on a
blank, start with LEFT and only switch to a plain join once you're sure
you want to lose that row on purpose.

Every one of these has the same shape. Something in the data is blank,
or two rows tie, and the ordinary-sounding question you asked gets
answered in a way that's technically consistent but not what you meant.
None of it shows up as an error. It shows up as a number that's wrong in
a way nobody notices until later.

More worked examples like these, free to browse with no signup, are at
the https://www.prepnplaced.com/prepnplaced-notes.