Spark Is a Smart Engine. So Why Doesn't It Cache Automatically?

Mohamed Said — Mon, 21 Sep 2026 12:23:16 +0000

How Is Executor Memory Organized?
Executor Memory is divided into three regions:
Execution Memory
Execution memory is the region where operations like joins, shuffle, sort, aggregation, etc. are computed.
Storage Memory
Used for caching user data structures and partitions derived from DataFrames.
Reserved Memory (300 MB)
A small, fixed slice (300MB) set aside specifically to guard against out-of-memory errors, kept untouchable by either of the other two regions.

Why This Memory Layout? Why Not Leave It as a Large Unbounded Pool?
Two different kinds of jobs can compete for this memory: Job A doing a computation job - sorting, joining, etc. - and Job B caching a large DataFrame. Without a bounded memory, this could lead us to starving a computing job, or spilling the cached DataFrame to disk, or forcing the active computing job to fail in the middle of a job. So this boundary saves us from situations like these.
But note: Execution Memory can use the storage area if it's not used, and vice versa, so there is flexibility in the design.
Executor Memory
Now that we understand how the memory layout of Spark executors is organized by default, let's move to the second important point, which is Spark Laziness.
Spark Laziness is considered one of the most important things in Spark, as Spark doesn't directly compute the result until an action forces it to compute the result. Instead, it builds the recipe, like saying, "If someone asks for this DataFrame, I have the way to build it":
df.filter(...).select(...).groupBy(...)
A chain of transformations like the one above is not executed directly; Spark builds the lineage, and one of the most important reasons it does that is to build the execution plan from this lineage, which the Catalyst Optimizer uses to optimize the query.
And with this lineage stored, Spark now doesn't need to save the intermediate results of the transformation process, as it knows how to recompute and get this DataFrame or the partition in case it needs it again for any reason.
Why Not Cache the Result by Default if It Will Be Used Again?
For the reason mentioned above, Spark has the lineage, so there is no reason to carry the cost of storing these results.
Spark was designed in the first place to work with high-scale, big, and large data, so caching the result by default between each step would lead to faster-filled memory and suffering from maintenance issues and job failures.
DataFrames are immutable, which means throughout the computing of this code, df.filter(...).select(...).groupBy(...), each step produces a completely new DataFrame. Spark does this to be resilient in case of failure, so as not to end up with a corrupted DataFrame, and to be able to recompute the partition from the original DataFrame. So even if you cache the DataFrame in the middle of a transformation step, that doesn't make any sense, since it will not be used again during this chain.
Back to the memory layout: if Spark keeps caching intermediate results, it will end up filling the storage memory faster, and will end up with two scenarios - if the execution memory is full, this will lead to OOM, or if it's available to borrow from it, continued caching will fill it too, which will cause later eviction of these cached partitions, or job failure, and a queue filled with lots of pending tasks.

So Spark's decision here is to leave the decision of caching to the developer.
So it's important to cache the results when caching will make an actual performance improvement, such as in cases of:
ML training (the exact same DataFrame gets scanned repeatedly across many training iterations).
Shared/reused DataFrames in a pipeline (multiple downstream reports or transformations all branching off the same intermediate result).

DEV Community: Mohamed Said

Spark Is a Smart Engine. So Why Doesn't It Cache Automatically?