PySpark Interview Questions: explaining how Spark runs your code

PySpark Interview Questions: explaining how Spark runs your code

A PySpark interview is not a quiz on API syntax. It tests three things: whether you understand how Spark runs your code, whether you can write DataFrame code live, and whether you can tune a job that is too slow. For senior roles there is a fourth — the scenario question, "a job that used to take ten minutes suddenly takes two hours; what do you do?" In every case the answer that lands is a short, confident sentence you say out loud, with the deeper detail ready for the follow-up.

This guide covers the PySpark interview questions that come up most in 2026: lazy evaluation and the shuffle, RDD versus DataFrame, the joins and data-skew problems that decide the round, and the optimisation levers interviewers probe. It is data engineering prep for a topic where explaining why Spark behaves the way it does matters as much as writing the transformation.

How Spark runs your code

Everything in a PySpark interview builds on one mental model. Your driver program builds a plan — a directed acyclic graph of steps — and executors run the actual work across the cluster in parallel. Two concepts carry most of the questions:

• Lazy evaluation. Transformations like select, filter and join do not run when you call them; they just add to the plan. Nothing executes until an action — count, collect, write — forces Spark to run the whole graph. This is why a notebook cell "returns instantly" and then the write takes ten minutes.
• Narrow vs wide transformations. A narrow transformation (map, filter, select) works within each partition and needs no data movement. A wide transformation (groupBy, join, distinct, repartition) requires a shuffle — moving data across the network to regroup it by key. Shuffles are the main cost in almost every Spark job, so knowing which operations trigger one is the foundation for every performance answer.

RDD vs DataFrame (and when to drop to an RDD)

The DataFrame API is the default for a reason: it runs through the Catalyst optimiser and the Tungsten execution engine, so Spark can rewrite and optimise your query before running it. An RDD is the lower-level abstraction with no optimiser — you reach for it only when your data does not fit neatly into columns or you need fine-grained control over partitioning that the DataFrame API cannot express. (The typed Dataset API is a JVM feature for Scala and Java, not something you use from PySpark.) The right interview answer is: "DataFrame by default for the Catalyst optimisation; RDD only for unstructured data or when I need manual control."

Joins and data skew: the make-or-break round

Joins are where PySpark interviews are won and lost, because a join is a wide transformation and the wrong strategy turns a fast job into a stalled one. The single most valuable optimisation is the broadcast join: when one side is small enough, Spark ships it to every executor so the join happens locally with no shuffle.

# broadcast the small table to avoid a shuffle join (the #1 PySpark speed-up)
from pyspark.sql.functions import broadcast
result = large_df.join(broadcast(small_df), "key")

When both sides are large, Spark falls back to a sort-merge join that shuffles both. The follow-up is almost always data skew: if one key holds far more rows than the others, a single task inherits a giant partition and runs long after every other task has finished. Name the fixes — salting the skewed key to spread it across partitions, letting Adaptive Query Execution split the skewed partition, or broadcasting the smaller side to avoid the shuffle entirely.

The optimisation levers interviewers probe

Performance questions come down to a handful of levers, and the common thread is reducing or balancing the shuffle:

• Broadcast small tables instead of shuffling them — the biggest single win.
• repartition vs coalesce. coalesce reduces the number of partitions without a full shuffle (cheap, for shrinking after a filter); repartition does a full shuffle and can increase partitions or rebalance skew (expensive, but evens things out).
• Cache or persist only when you reuse a DataFrame; caching something read once just wastes memory.
• Avoid Python UDFs. They serialise data row by row between the JVM and Python; prefer built-in functions or vectorised pandas UDFs.
• Lean on partition pruning and predicate pushdown so Spark reads only the files it needs, and let AQE (Adaptive Query Execution, a default since Spark 3) re-optimise at runtime.

The senior question ties them together: "a job that took ten minutes now takes two hours — how do you investigate?" Walk the checklist out loud — has data volume grown, is there new skew, is the shuffle spilling to disk, did a once-small table grow past the broadcast threshold, is there a small-files problem, is the partition count wrong? A staple live-coding task sits alongside it, "keep the latest row per customer", which is a window function:

# "latest row per customer" with a window function
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number, col
w = Window.partitionBy("customer_id").orderBy(col("event_ts").desc())
latest = df.withColumn("rn", row_number().over(w)).filter(col("rn") == 1).drop("rn")

Each of these is read on the four dimensions a scored round grades: problem solving (the right join and partition strategy), communication (explaining why the job shuffled or slowed), technical execution (correct DataFrame code), and time management (reaching a working, tuned answer in the round). Reading a question bank trains none of them.

Data engineering
Company-agnostic · PySpark / data processing · 60 min
Explain a Spark job out loud and diagnose why it got slow

Practise on Zynter →

How to prepare for a PySpark interview

The format rewards the short-answer-then-detail habit, so practise answering that way.

• Say the one-liner first, then the follow-up. "Broadcast join — ship the small table to every executor, no shuffle." Then explain the threshold if asked.
• Rehearse the diagnosis out loud. Take "the job got slow" and walk skew, spill, small files, partition count — as a spoken checklist, not a silent mental one.
• Know broadcast and repartition cold. They are the two levers that come up in almost every performance question.
• Drill DataFrame code live. Window functions, groupBy aggregations, joins — written quickly while you narrate.
• Build the shuffle intuition. For any transformation, be able to say instantly whether it shuffles.

For a deep bank of PySpark questions with worked DataFrame, RDD and optimisation answers, the sister platform PipeCode has a dedicated PySpark interview guide; use it to drill the mechanics, then come back here to rehearse explaining them out loud under follow-ups.

Data engineering
Company-agnostic · PySpark / data processing · 60 min
Rehearse broadcast joins and data-skew fixes with follow-ups

Practise on Zynter →

Frequently asked questions

What is lazy evaluation in Spark?

Lazy evaluation means transformations do not execute when you call them — they only build up a plan (a directed acyclic graph). Nothing actually runs until you call an action like count, collect or write, at which point Spark executes the whole graph at once. This lets the Catalyst optimiser rewrite and combine steps before any work happens, which is why a chain of select and filter returns instantly while the final write does all the real computation. Being able to explain why the write is slow and the transformations were not is a common follow-up.

What's the difference between narrow and wide transformations?

A narrow transformation — map, filter, select — operates within each partition and needs no data movement, so it is cheap. A wide transformation — groupBy, join, distinct, repartition — needs to regroup data by key, which forces a shuffle: moving data across the network between executors. Shuffles are the main performance cost in Spark, so interviewers want you to know instantly which operations trigger one and how to minimise them. Naming the shuffle as the cost centre is the heart of most optimisation answers.

When should you use a broadcast join?

Use a broadcast join when one side of the join is small enough to fit in each executor's memory. Spark ships that small table to every executor so the join happens locally, avoiding the shuffle that a normal join would require — usually the single biggest speed-up available on a DataFrame. The caveat is size: if the "small" table grows past the broadcast threshold, the broadcast can cause memory pressure or silently stop happening, which is a classic reason a once-fast job becomes slow. For two genuinely large tables, a sort-merge join is the right choice.

How do you handle data skew in PySpark?

Data skew is when one key holds far more rows than the others, so a single task inherits a huge partition and runs long after the rest finish. The common fixes are salting — adding a random suffix to the skewed key so its rows spread across multiple partitions, then aggregating back — and letting Adaptive Query Execution detect and split the skewed partition automatically. If one side of a skewed join is small, broadcasting it avoids the shuffle altogether. In an interview, naming the skew as the cause before reaching for a fix is the signal they want.

Repartition or coalesce — which should I use?

Use coalesce to reduce the number of partitions without a full shuffle — for example, after a heavy filter leaves you with many nearly empty partitions. Use repartition when you need a full shuffle to increase the partition count or to rebalance skewed data evenly across partitions. The trade-off is cost: coalesce is cheap because it just merges partitions, while repartition moves all the data. Saying "coalesce to shrink cheaply, repartition when I need to rebalance or grow" is the clean one-line answer.

How do I practise PySpark interview questions out loud?

Knowing the answers is not the same as explaining a DAG or diagnosing a slow job while someone pushes back. On Zynter, the Data engineering track runs live voice rounds where you write PySpark and SQL against real data and a voice interviewer follows up on your join strategy and your shuffle reasoning, then gives a debrief on problem solving, communication, technical execution and time management. You get a genuine data-engineer mock interview on demand, and the score is a practice signal, not a hiring decision.

Practise it on Zynter

You can memorise every Spark answer. The round is won explaining why the job shuffled — out loud.

Write PySpark and SQL in a real editor, then talk through your join strategy and why a job got slow with a voice interviewer that keeps asking "narrow or wide?" and "would you broadcast here?" — and read the debrief. Zynter.ai is on-demand mock interviews modelled on real company loops, with 150+ rounds across 20 companies (Sep 2026) — about $8–$12 for a full 60-minute mock, and two free credits to start.

Practise a data engineering round →
Browse all interview tracks