Start with the anti-pattern: repeated wide shuffles caused by repartitioning or doing joins followed immediately by groupBy on different keys. That’s the cheap, invisible way a cluster will chew CPU, network and spill to disk for minutes at a time.
A few concrete things that stop the bleeding fast: prefer a broadcast join when one side is small, pre-partition data on the join/group key so work stays local, and avoid unnecessary repartition()/coalesce() calls that reshuffle everything without reason. Also check spark.sql.shuffle.partitions — the default is often far higher than your actual parallelism needs.
How I debug this in practice: open the Spark UI, look at the stages with the biggest Shuffle Read/Write and the highest task spill. If a small number of partitions handle most bytes, you’ve got skew or a hot-shuffle hotspot. If many tasks are short but network reads are huge, it’s a repartitioning problem.
Quick checklist you can run through:
- try a broadcast join or lower spark.sql.autoBroadcastJoinThreshold for small tables
- set spark.sql.shuffle.partitions to match actual parallelism and enable adaptive execution (spark.sql.adaptive.enabled)
- pre-partition on stable keys and avoid redundant repartition()/repartitionByRange()
- cache only when it prevents repeated expensive shuffles, and unpersist promptly
If you want hands-on exercises and walkthroughs for cluster sizing, Spark configs and shuffle debugging, the Databricks training covers these topics with real notebooks and examples.
Top comments (0)