Spark Is Lazy, and That One Fact Explains Most Beginner Confusion

You write five lines of PySpark. You hit run. It finishes in a blink.

You think, "wow, that was fast."

Then you add .show() at the end, and suddenly everything takes ten seconds. Or worse, an error appears on a line that looks completely fine.

If this has happened to you, the reason is one idea: Spark is lazy. It doesn't do work when you write a line. It does work when you ask for a result.

Once this clicks, a lot of "weird" Spark behaviour stops being weird. Let's see it with real output.

I ran everything below with PySpark 4.3.0 in local mode (Java 17, Python 3.14). Nothing fancy, just pip install pyspark.

The Zomato cart

Think of ordering food on an app.

You add a biryani to the cart. You add a raita. You remove the raita. Nothing is cooked yet. You're only building a list of what you want.

Then you press "Place order". Now the kitchen actually starts.

Spark works the same way:

• Transformations are the cart. filter, select, withColumn, join, groupBy. They describe what you want and return immediately.
• Actions are "Place order". show, count, collect, take, write. These make Spark actually run.

The data

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.master("local[2]").appName("lazy").getOrCreate()

orders = spark.createDataFrame(
    [(1, "paid", 500), (2, "paid", 1200), (3, "refunded", 300),
     (4, "paid", 800), (5, "pending", 150)],
    ["order_id", "status", "amount"],
)

Five orders. Nothing special.

Experiment 1: transformations do no work

paid = orders.filter(F.col("status") == "paid")
big = paid.withColumn("amount_with_gst", F.col("amount") * 1.18)

Two lines. No output, no table, no data processed. big is not a result. It's a plan.

You can even ask Spark to show the plan, still without running anything:

big.explain()

== Physical Plan ==
*(1) Project [order_id#0L, status#1, amount#2L, (cast(amount#2L as double) * 1.18) AS amount_with_gst#3]
+- *(1) Filter (isnotnull(status#1) AND (status#1 = paid))
   +- *(1) Scan ExistingRDD[order_id#0L,status#1,amount#2L]

Read it from the bottom up. Scan the data, filter to paid, then add the GST column. That's your cart.

Now the action:

big.show()

+--------+------+------+---------------+
|order_id|status|amount|amount_with_gst|
+--------+------+------+---------------+
|       1|  paid|   500|          590.0|
|       2|  paid|  1200|         1416.0|
|       4|  paid|   800|          944.0|
+--------+------+------+---------------+

Only now did Spark actually touch the data.

Why would anyone design it this way?

Because Spark gets to see your whole plan before running any of it.

Look at the plan above. Your two separate lines (filter and withColumn) were combined into a single stage, marked *(1). Spark didn't run your filter, wait, then run your column step, wait again. It planned the whole thing as one pass.

If it ran each line the moment you wrote it, it couldn't do that. On a few rows you won't notice. On billions of rows, the difference between one pass and three is a big deal.

Experiment 2: every action starts over

Here's the part that surprises people. Each action recomputes the plan from the start.

To prove it, I'll count how many times Spark processes a row. I use an accumulator, which is a counter that Spark workers can add to, and a function that bumps it every time it runs.

from pyspark.sql.types import IntegerType

acc = spark.sparkContext.accumulator(0)

def double_it(x):
    acc.add(1)
    return x * 2

d = orders.withColumn("double", F.udf(double_it, IntegerType())("amount"))

print("before:", acc.value)
d.agg(F.sum("double")).collect()
print("after action 1:", acc.value)
d.agg(F.sum("double")).collect()
print("after action 2:", acc.value)

before: 0
after action 1: 5
after action 2: 10

Look at the three numbers.

• 0 after defining d. Building the DataFrame ran nothing.
• 5 after the first action. Five rows, five calls.
• 10 after the second action. Spark did the same work again.

Spark didn't remember the result. It has no idea you plan to use it twice. It only has the recipe.

Now with cache():

acc2 = spark.sparkContext.accumulator(0)

def double_it2(x):
    acc2.add(1)
    return x * 2

d2 = orders.withColumn("double", F.udf(double_it2, IntegerType())("amount")).cache()

d2.agg(F.sum("double")).collect()
print("cached, after action 1:", acc2.value)
d2.agg(F.sum("double")).collect()
print("cached, after action 2:", acc2.value)

cached, after action 1: 5
cached, after action 2: 5

The second action reused the stored result. Still 5.

A small honesty note: accumulators inside transformations are fine for a demo like this, but don't build real logic on them. If Spark retries a task, the count can go up.

So when should you cache? When one DataFrame feeds more than one action and the work to build it is expensive. If you use it once, cache just wastes memory.

Experiment 3: errors show up late

This is the one that wastes the most debugging time. I'll add a function that fails on one value:

def bad(x):
    return int("abc") if x == 300 else x

broken = orders.withColumn("fixed", F.udf(bad, IntegerType())("amount"))
print("built broken df ok, no error yet")

built broken df ok, no error yet

No error. The bug is in the plan, but nothing has run. Now:

broken.show()

PythonException: ... ValueError: invalid literal for int() with base 10: 'abc'

The failure appears at .show(). The bug was a line you wrote earlier. In a long notebook, the error line and the cause can be many cells apart.

Not all mistakes wait, though. Spark checks column names when you build the plan:

orders.select("nope")

AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function
parameter with name `nope` cannot be resolved. Did you mean one of the following?
[`amount`, `order_id`, `status`].

That one fails immediately, even though there is no action. Spark can check names and types without reading the data. What it can't know until it runs is what's inside your rows or what your Python functions do with them.

Rule of thumb: a typo fails early. A bad value fails late.

Common mistakes

1. Sprinkling count() to debug. Each count() is an action. Each one recomputes the whole plan. Ten of them can mean ten full runs. If you're debugging, .limit(20).show() or one cache() first is kinder to the cluster.

2. Timing the wrong line. "This join is slow." No. The join returned instantly. The write at the end paid for everything. Time actions, not transformations.

3. collect() on big data. It's an action that pulls everything to one machine. On five rows, fine. On millions, you'll run out of memory.

4. Caching everything. Cache only what you reuse. Call unpersist() when you're done.

The interview version

If someone asks "What is lazy evaluation in Spark?", you can say:

"Transformations like filter and select don't run immediately. Spark just records them as a plan. Work happens when I call an action like show, count or write. That lets Spark optimise the whole plan together, but it also means each action recomputes from the start unless I cache."

That's a solid answer, and now you can back it up with an example.

Quick recap

• Transformations build a plan. Actions run it.
• explain() shows the plan without running anything.
• Every action recomputes unless you cache().
• Name typos fail early. Bad data fails late.

Try it yourself

If you'd like to practise DataFrame operations after this, I've put some PySpark problems on LakeBench, a platform I'm building: lakebench.in/pyspark-practice. The free ones are the beginner set.

Quick question for you. What's the longest you've waited on a Spark job, only to find out the bug was three cells up?