Google Cloud Dataproc & Dataproc Serverless: Managed Spark on GCP

google cloud dataproc is the managed Hadoop and Spark service on Google Cloud — you get the open-source engines you already know (Spark, Hive, Trino, Flink, PySpark) running on a cluster that Google provisions in about ninety seconds, and you throw the cluster away when the job finishes. It is not a rewrite of Spark, not a proprietary dialect, and not a UI you click through. You call gcloud dataproc clusters create, submit a job, and the same PySpark or Spark SQL that runs on your laptop runs on managed YARN, reading from and writing to Google Cloud Storage.

That is a very different shape from the way teams ran Spark before it: a long-lived Hadoop cluster you patched, resized, and paid for around the clock, or a hand-built fleet of Compute Engine VMs you turned into a Spark cluster with Ansible and hope. This guide walks through the five ideas an interviewer will actually probe — why a managed, disposable cluster beats a standing one, the master / primary-worker / secondary-worker anatomy, the ephemeral-cluster and autoscaling cost pattern, storage decoupling through the GCS connector plus Dataproc Metastore and the BigQuery connector, and Dataproc Serverless for Spark — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the Spark SQL practice library →, rehearse the transformation logic on the PySpark practice set →, and tune your jobs on the optimization practice set →.

---

On this page

• Why Dataproc changes Spark on GCP in 2026
• Cluster anatomy — masters, primary & secondary workers
• Ephemeral clusters, autoscaling & the cost pattern
• Storage & metadata — GCS connector, Metastore, BigQuery
• Dataproc Serverless for Spark — batch without clusters
• Cheat sheet — Dataproc recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why Dataproc changes Spark on GCP in 2026

Dataproc is a managed cluster you rent by the job, not a platform you operate all year — that one fact decides where it fits

The one-sentence invariant: Dataproc runs stock open-source Spark and Hadoop on infrastructure Google provisions and tears down on demand, so a cluster becomes a per-job resource rather than a standing system you keep alive. Everything that makes Dataproc attractive to a data engineering team follows from that. There is no cluster to patch on a maintenance window, no capacity you pay for while it sits idle overnight, and no bespoke bootstrap tooling; a Dataproc cluster starts in roughly ninety seconds and can be deleted the moment the last job commits.

What Dataproc gives you and what it deliberately leaves to open source.

• Managed operations. Google provisions the VMs, installs a versioned image (a pinned set of Spark, Hadoop, Hive, and connector versions), wires YARN and HDFS, and exposes cluster/job APIs. You do not build the AMI or configure the ResourceManager.
• Stock engines. The Spark, PySpark, Spark SQL, Hive, Trino/Presto, and Flink you run are the real open-source projects, not a fork. Code is portable off Dataproc unchanged.
• Storage is not baked in. Dataproc gives every cluster ephemeral HDFS on the workers' disks, but the durable layer is Google Cloud Storage (gs://). Keeping your data out of the cluster is what lets the cluster be disposable.

Where Dataproc sits against the alternatives.

• vs self-managed Spark on Compute Engine. Rolling your own cluster on raw VMs means you own the image, the scaling, the autohealing, and the upgrades. Dataproc hands all of that to Google while keeping the same Spark, so you trade nothing in portability and lose the operational tax.
• vs Dataflow. Dataflow runs Apache Beam pipelines with a fully serverless, auto-tuning runner and no cluster concept at all. Choose Dataflow for greenfield streaming/batch in Beam; choose Dataproc when you already have Spark/Hadoop/Hive code and want to lift it to GCP with minimal change.
• vs BigQuery. BigQuery is a serverless SQL warehouse — reach for it when the work is SQL over tables. Dataproc is for Spark-shaped work (custom Python, ML feature pipelines, complex non-SQL transforms) and often reads from and writes to BigQuery through a connector.

What interviewers listen for.

• Do you say "Dataproc is managed open-source Spark, not a proprietary engine" in the first sentence? — senior signal.
• Do you frame the cluster as "ephemeral and job-scoped, with data in GCS" unprompted? — the whole cost story.
• Do you place Dataproc against Dataflow and BigQuery by workload shape rather than calling one strictly better? — required framing.
• Do you mention Dataproc Serverless when the answer is "I don't even want to size a cluster"? — 2026 signal.

Worked example — a cluster and a job in three commands

Detailed explanation. The canonical Dataproc "hello world" creates a small cluster, submits a PySpark job that reads and writes GCS, and deletes the cluster — the whole ephemeral pattern in three commands. It looks trivial, and that is the point: the same three commands that run a toy word-count scale unchanged to a fifty-node job, because Dataproc only ever sees "a cluster spec, a job, and a delete."

Question. Run a PySpark job that reads gs://my-bucket/in/ and writes gs://my-bucket/out/, without leaving a cluster running afterward.

Input.

resource	value
region	us-central1
cluster	etl-eph (2 workers)
job	wordcount.py (reads/writes gs://)

Code.

gcloud dataproc clusters create etl-eph \
  --region=us-central1 --num-workers=2 --max-idle=10m    # (1) create + arm auto-delete

gcloud dataproc jobs submit pyspark gs://my-bucket/code/wordcount.py \
  --cluster=etl-eph --region=us-central1 \
  -- gs://my-bucket/in/ gs://my-bucket/out/              # (2) submit the PySpark job

gcloud dataproc clusters delete etl-eph --region=us-central1   # (3) delete now (or via TTL)

Step-by-step explanation. clusters create provisions one master and two worker VMs, installs the Dataproc image, and starts YARN and HDFS; --max-idle=10m arms an auto-delete timer. jobs submit pyspark uploads (or references) the script, submits it to YARN through the Dataproc jobs API, and streams driver logs back to you; the -- passes script arguments. Because the script reads and writes gs://, nothing durable lives on the cluster. clusters delete releases the VMs — and even if you forget, --max-idle deletes the cluster ten minutes after the last job finishes.

Output.

stage	result
create	cluster etl-eph RUNNING in ~90s (1 master + 2 workers)
submit	job state DONE, output written to gs://my-bucket/out/
delete	cluster removed; you pay only for the minutes it ran

Rule of thumb. If your data lives in gs:// and your job is a script, the cluster is disposable — the toy example and the production job differ only in size and code, never in the lifecycle.

---

2. Cluster anatomy — masters, primary & secondary workers

master, primary workers, and secondary workers are the entire cluster mental model — learn these three roles and sizing stops being guesswork

A Dataproc cluster has exactly three node roles, and an interviewer who asks "walk me through a Dataproc cluster" wants these three in order, with the storage implication of each. Get the roles crisp and preemptible-VM sizing stops being scary.

The three node roles.

• Master — schedules and coordinates. The master runs the YARN ResourceManager and the HDFS NameNode (plus the Spark History Server and job drivers in cluster mode). A standard cluster has one master; High Availability mode runs three masters so a single failure does not take the cluster down.
• Primary workers — compute plus HDFS. Each primary worker runs a YARN NodeManager (to execute Spark tasks) and an HDFS DataNode (to store blocks). Primary workers are standard, non-preemptible VMs by default because they hold HDFS data — losing one loses blocks.
• Secondary workers — compute only. Secondary workers run a NodeManager but no DataNode, so they add task capacity without storing any HDFS data. Because they hold nothing durable, they are the right place for Spot (preemptible) VMs, which Google can reclaim at any time.

Why the primary/secondary split exists.

• Spot VMs are cheap but reclaimable. A Spot worker can vanish with ~30 seconds' notice, and preemptible instances have at most a 24-hour life. That is fine for stateless compute and disastrous for HDFS blocks — so Dataproc structurally keeps HDFS on primary workers and lets Spot ride on secondary workers only.
• Two knobs, two purposes. --num-workers sizes durable primary capacity; --num-secondary-workers (with --secondary-worker-type=spot) sizes cheap, elastic compute. Interviewers love this because getting it backwards — Spot on primaries — is a data-loss bug.

Initialization actions — customizing every node at create.

• What they are. Shell scripts stored in GCS that Dataproc runs on every node during cluster creation, before the cluster reports ready. Use them to install a Python package, a monitoring agent, or a JDBC driver.
• Where they run. On all masters and all workers; you can gate behavior with the ROLE metadata (Master vs Worker) so a script does one thing on the master and another on workers.
• Cost of getting it wrong. A slow or failing init action delays or fails the whole cluster create, so keep them idempotent and fast.

Worked example — a cluster with primary and Spot secondary workers

Detailed explanation. Real cost-tuned clusters split durable and elastic capacity. Here a cluster keeps two standard primary workers (for HDFS and the application master) and adds four Spot secondary workers for cheap extra executors. If Google reclaims a Spot node mid-job, YARN reschedules its tasks on surviving nodes — the job slows, it does not corrupt.

Question. Create a cluster with 2 standard primary workers and 4 Spot secondary workers, and explain what each role stores.

Input. One cluster spec; the goal is cheap compute without risking HDFS data.

Code.

gcloud dataproc clusters create mixed \
  --region=us-central1 \
  --num-workers=2 \
  --num-secondary-workers=4 \
  --secondary-worker-type=spot \
  --initialization-actions=gs://my-bucket/init/pip-install.sh

Step-by-step explanation. --num-workers=2 creates two primary workers, each a NodeManager + DataNode on standard VMs, so HDFS blocks and the Spark application master land on stable nodes. --num-secondary-workers=4 --secondary-worker-type=spot adds four secondary workers that are NodeManager-only Spot VMs — pure compute, no HDFS. The init action runs pip-install.sh on every node so all six workers have the same Python libraries. Total task capacity is six workers at a fraction of six standard-worker cost, and a Spot reclamation only removes compute.

Output.

node role	count	runs	VM type	stores HDFS?
master	1	ResourceManager + NameNode	standard	yes (NameNode metadata)
primary worker	2	NodeManager + DataNode	standard	yes (blocks)
secondary worker	4	NodeManager only	Spot	no

Rule of thumb. Put durable things (HDFS, the app master) on primary workers, put cheap elastic compute on Spot secondary workers, and never invert that — Spot on a primary worker is a data-loss waiting to happen.

Dataproc interview question on preemptible-worker safety

Question. A teammate sets --num-workers=1 --num-secondary-workers=40 --secondary-worker-type=spot to save money. Jobs start failing intermittently with lost-executor and shuffle-fetch errors whenever Spot nodes are reclaimed. What is wrong and how do you make the cluster both cheap and reliable?

Solution Using a primary-worker floor plus Enhanced Flexibility Mode

Code.

gcloud dataproc clusters create resilient \
  --region=us-central1 \
  --num-workers=4 \
  --num-secondary-workers=40 \
  --secondary-worker-type=spot \
  --properties=dataproc:efm.spark.shuffle=primary-worker

Step-by-step trace.

symptom	root cause	fix applied
lost executors when Spot reclaimed	too few durable workers; HDFS + AM under-provisioned	raise --num-workers to a real floor (4)
shuffle-fetch failures	shuffle blocks lived on reclaimed Spot nodes	EFM writes shuffle to primary workers
whole job retries	app master ran on a Spot node	primaries host the AM, which now survives

1. With only one primary worker, HDFS, the application master, and shuffle data were dangerously concentrated; every Spot reclamation could kill something the job depended on.
2. Raising --num-workers to 4 gives a durable floor for HDFS blocks and the Spark application master, so reclaiming a Spot node removes only replaceable compute.
3. Enhanced Flexibility Mode (efm.spark.shuffle=primary-worker) moves shuffle output onto the non-preemptible primary workers, so a reclaimed Spot executor no longer takes shuffle blocks with it — the classic cause of cascading fetch failures.
4. The 40 Spot secondary workers still provide the bulk of cheap task capacity; they are now purely additive, so the cluster stays cheap and stops failing.

Output:

metric	before	after
job success under preemption	flaky	stable (tasks reschedule)
shuffle data on Spot nodes	yes (fragile)	no (on primaries)
cost	lowest, unreliable	slightly higher, reliable

Why this works — concept by concept:

• Primary/secondary split — durable state (HDFS, AM, with EFM the shuffle) lives on standard primary workers; Spot secondary workers carry only reschedulable compute.
• Enhanced Flexibility Mode — EFM decouples shuffle from executor lifetime by parking shuffle on primaries, which is what makes aggressive Spot ratios safe.
• Preemption tolerance — YARN reschedules tasks from a reclaimed node onto survivors, so losing Spot capacity degrades throughput instead of failing the job.
• Cost vs reliability dial — the primary floor and Spot ratio are two independent knobs; you tune the trade-off rather than accept an all-Spot cluster that saves money until it doesn't.
• Cost — money is roughly O(primary workers at on-demand + secondary workers at Spot price); reliability scales with the primary floor, not the Spot count.

Spark SQL
Topic — spark-sql
Spark SQL query and aggregation problems

Practice →

PySpark Topic — pyspark PySpark transformation and job-shape problems

Practice →

---

3. Ephemeral clusters, autoscaling & the cost pattern

Spin a cluster per job, autoscale it while it runs, and delete it when it idles — the ephemeral pattern is Dataproc's whole cost story

A standing Hadoop cluster bills you 24/7 for capacity you use a few hours a night. The Dataproc idiom flips that: create a cluster scoped to a job (or a batch of jobs), let it autoscale to the work, and delete it the moment it goes idle, so you pay for compute-minutes, not for a cluster that naps. The thing that makes this safe is that your data is already in GCS — deleting the cluster loses nothing.

The ephemeral lifecycle.

• Create → run → delete. Provision a cluster, submit the job(s), tear it down. For a single job this is three commands; for a DAG of jobs it is a Workflow Template, which creates a managed (or ephemeral) cluster, runs an ordered set of jobs, and deletes the cluster automatically.
• TTL auto-delete. --max-idle deletes the cluster after a period with no running jobs; --max-age caps absolute lifetime; --delete-max-idle / scheduled deletion guard against a forgotten cluster. These are the cheap insurance that a crashed pipeline does not leave VMs running for a weekend.
• Why it is cheap. No idle capacity, no over-provisioning "just in case," and Spot secondary workers on top — the cluster's cost tracks the job, not the calendar.

Autoscaling policies — sizing the cluster to the work while it runs.

• What scales. An autoscaling policy sets minInstances / maxInstances for secondary workers (and optionally primary), and Dataproc adds or removes workers based on YARN pending and available memory — more pending containers, scale up.
• The knobs. scaleUpFactor and scaleDownFactor control how aggressively it grows and shrinks; cooldownPeriod prevents thrashing between decisions; gracefulDecommissionTimeout lets a node finish its tasks (and hand off shuffle) before removal.
• Graceful decommission is the safety valve. Without it, scaling down mid-shuffle kills tasks and forces retries; with it, Dataproc drains a worker before reclaiming it.

Failure modes interviewers probe.

• Scale-down that never happens — if gracefulDecommissionTimeout is huge or long tasks hold nodes, the cluster stays large; tune the timeout and prefer scaling secondary workers.
• Thrashing — a too-short cooldownPeriod makes the cluster flap up and down; widen the cooldown so each decision has time to take effect.

Worked example — an ephemeral autoscaling cluster

Detailed explanation. The everyday cost-tuned pattern is an ephemeral cluster that attaches an autoscaling policy at create and arms an idle TTL. The policy grows secondary Spot workers when YARN has pending containers and shrinks them (gracefully) when demand drops; the TTL guarantees the cluster cannot outlive its usefulness.

Question. Create a job-scoped cluster that starts at 2 secondary workers, scales up to 10 under load, decommissions gracefully, and auto-deletes after 30 minutes idle.

Input. An autoscaling policy plus a cluster referencing it.

Code.

workerConfig: { minInstances: 2, maxInstances: 2 }              # policy.yaml — fixed primaries
secondaryWorkerConfig: { minInstances: 2, maxInstances: 10 }
basicAlgorithm:
  cooldownPeriod: 2m
  yarnConfig:
    scaleUpFactor: 0.5
    scaleDownFactor: 1.0
    gracefulDecommissionTimeout: 1h

gcloud dataproc autoscaling-policies import etl-policy \
  --region=us-central1 --source=policy.yaml

gcloud dataproc clusters create etl-auto \
  --region=us-central1 \
  --num-workers=2 --num-secondary-workers=2 --secondary-worker-type=spot \
  --autoscaling-policy=etl-policy \
  --max-idle=30m

Step-by-step explanation. The policy pins primary workers at 2 (HDFS stays stable) and lets secondary workers float between 2 and 10. When a job floods YARN with pending containers, scaleUpFactor=0.5 adds roughly half the shortfall each cooldown; when demand falls, scaleDownFactor=1.0 removes idle capacity, but gracefulDecommissionTimeout=1h first lets each departing worker finish its tasks. --max-idle=30m deletes the whole cluster half an hour after the last job ends, so nothing lingers.

Output.

phase	secondary workers	trigger
start	2	policy minimum
heavy stage	10	pending YARN memory high, scale up
cooldown	4	demand dropped, graceful scale down
30m idle	cluster deleted	--max-idle TTL

Rule of thumb. Autoscale the secondary (Spot) workers, keep primaries fixed for HDFS, always set gracefulDecommissionTimeout so scale-down never kills a running shuffle, and always set a TTL so a failed pipeline cannot bleak a cluster.

Dataproc interview question on scale-down correctness

Question. Your autoscaling cluster grows fine but nightly jobs randomly fail with "FetchFailed" whenever the cluster scales down. The team's fix was to disable autoscaling entirely, which doubled cost. What actually causes the failures and how do you keep autoscaling on safely?

Solution Using graceful decommission plus shuffle protection

Code.

basicAlgorithm:                          # corrected autoscaling policy (excerpt)
  cooldownPeriod: 4m
  yarnConfig:
    scaleUpFactor: 0.5
    scaleDownFactor: 0.5
    gracefulDecommissionTimeout: 1h      # let tasks + shuffle drain before removal
secondaryWorkerConfig:
  minInstances: 2
  maxInstances: 20                       # plus --properties=dataproc:efm.spark.shuffle=primary-worker

Step-by-step trace.

event	with abrupt scale-down	with graceful + EFM
worker chosen for removal	killed immediately	drained first (up to 1h)
shuffle blocks on it	lost → FetchFailed	on primaries (EFM), never lost
running tasks	retried elsewhere	finish before node leaves
job outcome	intermittent failure	completes; cluster shrinks after

1. Autoscaling removed secondary workers that still held shuffle output; downstream stages fetching that shuffle hit FetchFailed and the stage retried, sometimes exhausting retries.
2. Setting gracefulDecommissionTimeout=1h tells Dataproc to stop scheduling new work on a departing node and wait for its tasks (and shuffle serving) to finish before reclaiming it.
3. Enabling Enhanced Flexibility Mode parks shuffle on primary workers, so even an abruptly reclaimed Spot node carries no shuffle blocks — removing the failure at its source.
4. A slightly longer cooldownPeriod and gentler scaleDownFactor stop the cluster from shrinking so fast that it fights its own running stages.

Output:

metric	before	after
autoscaling	disabled (2x cost)	enabled, safe
FetchFailed on scale-down	frequent	eliminated
cost	high, flat	low, tracks load

Why this works — concept by concept:

• Graceful decommission — draining a node before removal converts "kill and retry" into "finish then release," which is what makes scale-down safe under shuffle-heavy jobs.
• Shuffle placement (EFM) — keeping shuffle off ephemeral workers means node loss never destroys data a later stage needs.
• Cooldown and factors — widening the cooldown and softening the scale-down factor stops thrashing so decisions settle before the next one fires.
• Autoscale the right pool — floating secondary workers while pinning primaries keeps HDFS stable while compute flexes.
• Cost — cost tracks O(pending YARN demand) instead of peak provisioning, with correctness bought by the decommission timeout rather than by turning autoscaling off.

Optimization
Topic — optimization
Cluster-sizing and cost-optimization problems

Practice →

PySpark Topic — pyspark Shuffle-aware PySpark job problems

Practice →

---

4. Storage & metadata — GCS connector, Metastore, BigQuery

Keep the data in gs:// and the metadata in the Metastore — decoupling storage from the cluster is what makes "delete the cluster" harmless

The reason you can throw a Dataproc cluster away is that nothing you care about lives on it. Source and output data sit in Google Cloud Storage, read and written through the GCS connector; table definitions sit in Dataproc Metastore; and BigQuery tables are reached through a connector. HDFS on the cluster is scratch space, not a home. Get this separation right and an ephemeral cluster is a pure compute lease.

The GCS connector — gs:// as a Hadoop filesystem.

• Drop-in HDFS replacement. The GCS connector ships in the Dataproc image and implements the Hadoop FileSystem API, so Spark reads gs://bucket/path exactly as it would hdfs:// — same spark.read.parquet(...), different scheme.
• Durable and decoupled. Because the data is in GCS, it survives every cluster delete, is shared across clusters and services, and separates storage cost from compute cost.
• Object-store semantics. GCS is object storage: listing large prefixes and rename-heavy commit protocols are slower than on HDFS, so favor the direct/S3A-style committers and partition-pruned reads. This is why HDFS still earns a role.

HDFS on Dataproc — scratch, not source of truth.

• What it is good for. Cluster-local HDFS (on worker disks) is fast, low-latency scratch for shuffle spill and multi-stage intermediates within a single job.
• What it must not be. It dies with the cluster and rides partly on Spot secondary workers, so never treat it as durable storage. Land inputs and outputs in gs://.

Dataproc Metastore and the BigQuery connector.

• Dataproc Metastore. A fully managed, highly available Hive Metastore service that lives outside any cluster. Point ephemeral clusters at one shared Metastore and your Hive/Spark SQL table definitions persist across cluster deletes and are shared by every cluster and by BigQuery — solving the "my tables vanished when the cluster went away" problem.
• BigQuery connector. The Spark BigQuery connector reads tables through the high-throughput BigQuery Storage API (parallel, columnar, predicate-pushed) and writes back via a GCS staging path or the Storage Write API — so Dataproc Spark and BigQuery interoperate without exporting CSVs by hand.

Worked example — read GCS, write BigQuery from PySpark

Detailed explanation. The everyday interoperability pattern reads Parquet from gs:// and writes an aggregate to a BigQuery table, using the two connectors that ship with Dataproc. No manual export, no CSV round-trip — the connectors handle the wire formats.

Question. From a Dataproc PySpark job, read gs://lake/events/ (Parquet), aggregate daily counts, and write them to the BigQuery table analytics.daily_counts.

Input.

{"gcs_input": "gs://lake/events/", "bq_output": "analytics.daily_counts",
 "temp_bucket": "gs://lake/tmp/"}

Code.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("gcs-to-bq").getOrCreate()

events = spark.read.parquet("gs://lake/events/")          # GCS connector

daily = (events
         .groupBy(F.to_date("event_ts").alias("day"))
         .count())

(daily.write.format("bigquery")                           # BigQuery connector
      .option("table", "analytics.daily_counts")
      .option("temporaryGcsBucket", "lake/tmp")           # GCS-staged write
      .mode("overwrite")
      .save())

Step-by-step explanation. spark.read.parquet("gs://...") uses the GCS connector to read columnar files directly from the bucket, pruning partitions and pushing down projections. The aggregation runs on YARN across the workers. write.format("bigquery") hands the result to the BigQuery connector, which stages Parquet into temporaryGcsBucket and loads it into analytics.daily_counts — overwrite truncates the destination first. Nothing durable touched cluster HDFS, so the cluster can be deleted right after.

Output.

step	reads/writes	mechanism
read events	gs://lake/events/	GCS connector (Parquet)
aggregate	in-cluster	Spark on YARN
write counts	analytics.daily_counts	BigQuery connector via GCS staging

Rule of thumb. Inputs and outputs belong in gs:// and BigQuery; HDFS is only for in-job scratch — that is what lets you delete the cluster the second the write commits.

Dataproc interview question on sharing metadata across ephemeral clusters

Question. Your team runs ten ephemeral clusters a day. Each one defines Hive external tables over gs:// in its local Metastore, and every morning analysts complain the tables "don't exist" on a fresh cluster. Data is fine in GCS; only the table definitions keep disappearing. How do you fix it?

Solution Using a shared Dataproc Metastore

Code.

gcloud metastore services create shared-hms \
  --location=us-central1 --tier=DEVELOPER      # (1) one shared managed Hive Metastore

gcloud dataproc clusters create eph-$(date +%s) \
  --region=us-central1 --max-idle=30m \
  --dataproc-metastore=projects/PROJ/locations/us-central1/services/shared-hms   # (2) attach it

-- tables are external over gs://, so data + metadata both outlive the cluster
CREATE EXTERNAL TABLE events (id BIGINT, event_ts TIMESTAMP)
STORED AS PARQUET LOCATION 'gs://lake/events/';

Step-by-step trace.

day-to-day event	with local Metastore	with shared Dataproc Metastore
cluster A creates events	stored in A's local HMS	stored in shared HMS
cluster A deleted	table definition gone	definition persists
cluster B (next morning)	events not found	events visible immediately
BigQuery / other tools	cannot see it	can read the same catalog

1. Each ephemeral cluster ran its own Hive Metastore on the master, so table definitions died with the cluster even though the underlying gs:// data was untouched.
2. Creating one Dataproc Metastore service moves the catalog outside any cluster into a managed, HA Hive Metastore.
3. Attaching every cluster with --dataproc-metastore=... makes them all read and write the same catalog, so a table created by one cluster is instantly visible to the next.
4. Because the tables are external over gs://, both the data (GCS) and the metadata (shared Metastore) survive every cluster delete — the ephemeral pattern finally holds end to end.

Output:

concern	outcome
table definitions	persist across cluster deletes
cross-cluster visibility	all clusters share one catalog
data location	unchanged in gs:// (external tables)

Why this works — concept by concept:

• Managed Metastore — lifting the Hive Metastore out of the cluster into a standalone managed service decouples catalog lifetime from cluster lifetime.
• External tables over gs:// — the data was always durable in GCS; only the metadata was ephemeral, so persisting the catalog completes the decoupling.
• Shared catalog — one Metastore behind many clusters (and BigQuery) gives a single source of truth for schema instead of ten divergent local copies.
• Ephemeral-safe — with data in GCS and metadata in the Metastore, deleting a cluster is a no-op for correctness, which is the entire point of ephemeral Dataproc.
• Cost — one small always-on Metastore is a fixed cost far below running a standing cluster just to hold table definitions.

Spark SQL
Topic — spark-sql
External-table and catalog Spark SQL problems

Practice →

Optimization Topic — optimization Storage-layout and read-pruning problems

Practice →

---

5. Dataproc Serverless for Spark — batch without clusters

gcloud dataproc batches submit runs a Spark job with no cluster to size, autoscale, or delete — the endgame of the ephemeral pattern

Ephemeral clusters already remove the standing bill, but you still choose machine types, worker counts, and a TTL. Dataproc Serverless for Spark removes even that: you submit a batch, and Google provisions a right-sized, autoscaling Spark runtime, runs your job, and tears it all down — you never create a cluster. Say it in one breath: Serverless is a cluster you don't provision, autoscaled per job, billed by the second.

The Serverless batch model.

• Submit a batch, not a cluster. gcloud dataproc batches submit pyspark|spark|spark-sql ... runs one workload. There is no cluster resource to create, size, or delete, and no idle capacity to worry about.
• Autoscaling by default. Serverless uses Spark dynamic allocation: it adds and removes executors to match the job's demand, so you don't guess --num-workers. You can cap it with executor/memory properties when you need determinism.
• Managed everything. Google picks the runtime version (a pinned Spark + connector bundle), applies your properties, streams logs to Cloud Logging, and cleans up on completion.

Billing and when it wins.

• How you pay. Serverless bills by Data Compute Units (DCU) per second plus shuffle-storage, with a one-minute minimum per batch — you pay for the work, not for a provisioned cluster.
• When Serverless beats a cluster. Bursty, independent batch jobs; teams that don't want to own sizing; spiky schedules where a standing cluster would idle. Each batch is isolated, so noisy neighbors and version conflicts disappear.
• When a cluster is still better. Many tiny, latency-sensitive jobs that benefit from a warm, always-on cluster; interactive exploration; workloads needing specific components (a long-running Trino/HBase service), custom daemons, or fine-grained machine control. Persistent clusters keep caches warm; Serverless pays cold-start per batch.

The same connectors carry over.

• GCS, BigQuery, Metastore all apply. A Serverless batch reads gs://, writes BigQuery through the connector, and can attach a Dataproc Metastore for a shared catalog — the storage/metadata decoupling from the previous section is exactly what makes clusterless batches practical.
• Bring your own deps. Package Python dependencies (a custom container image or --py-files/--jars) so the serverless runtime matches what your job needs.

Worked example — submit a Serverless PySpark batch

Detailed explanation. The everyday Serverless pattern submits a PySpark file as a batch, attaches a Metastore for the shared catalog, and lets dynamic allocation size the executors. There is no cluster create and no cluster delete — the batch is the whole unit of work.

Question. Run the same GCS-to-BigQuery aggregation as a Serverless batch, with no cluster, attaching the shared Metastore.

Input. A PySpark file in GCS plus a batch submit command.

Code.

gcloud dataproc batches submit pyspark gs://lake/code/gcs_to_bq.py \
  --region=us-central1 \
  --deps-bucket=gs://lake/deps/ \
  --metastore-service=projects/PROJ/locations/us-central1/services/shared-hms \
  --properties=spark.dynamicAllocation.enabled=true,\
spark.dynamicAllocation.maxExecutors=20

Step-by-step explanation. batches submit pyspark hands the script to the Serverless runtime — no clusters create anywhere. --metastore-service=... attaches the shared Dataproc Metastore so the job sees the same external tables a cluster would. Dynamic allocation starts with a few executors and scales toward maxExecutors=20 as the aggregation demands, then releases them. When the batch finishes, everything is torn down and you are billed DCU-seconds for exactly what ran.

Output.

aspect	cluster job	Serverless batch
provision cluster	yes (create + delete)	none
sizing	you pick workers	dynamic allocation
billing	cluster-minutes	DCU-seconds (1-min min)
cleanup	delete / TTL	automatic

Rule of thumb. For an independent batch that doesn't need a warm cluster, reach for batches submit first — it deletes the last two operational chores (sizing and teardown) that ephemeral clusters still leave you.

Dataproc interview question on choosing Serverless vs a cluster

Question. You have two workloads: (a) 200 tiny, latency-sensitive Spark SQL jobs per hour that share cached reference data, and (b) a handful of large, independent nightly ETL batches that spike then stop. Which runs on a persistent cluster and which on Serverless, and why?

Solution Using Serverless for bursty batches, a cluster for warm many-small

Code.

gcloud dataproc batches submit spark \
  --region=us-central1 --class=com.acme.NightlyEtl \
  --jars=gs://lake/code/etl.jar \
  --properties=spark.dynamicAllocation.maxExecutors=50   # (b) nightly ETL → Serverless

gcloud dataproc clusters create warm \
  --region=us-central1 --num-workers=4 \
  --num-secondary-workers=4 --secondary-worker-type=spot \
  --autoscaling-policy=warm-policy       # (a) many-small, cache-sharing → warm cluster

Step-by-step trace.

workload	shape	best runtime	reason
(b) nightly ETL	few, large, spiky, independent	Serverless	no idle between spikes; autoscaled per batch
(a) 200 jobs/hr	many, tiny, share cache, latency-sensitive	persistent cluster	warm executors + cached data; avoids per-batch cold start

1. Workload (b) is bursty and independent: a standing cluster would idle between nightly spikes, while Serverless spins up per batch, autoscales to 50 executors, and bills only the DCU-seconds used.
2. Workload (a) is many tiny jobs that reuse cached reference data: Serverless would pay cold-start and lose the cache on every batch, so a persistent, autoscaling cluster keeps executors and cache warm and answers each job fast.
3. The persistent cluster still uses Spot secondary workers and autoscaling to stay cheap during the day, and can be deleted overnight — it is "persistent" only across the busy window, not forever.
4. Both read gs:// and share the Metastore, so the storage/metadata decoupling lets you route each workload to the runtime that fits without changing the data layer.

Output:

workload	runtime chosen	cost/perf outcome
(b) nightly ETL	Serverless batches	no idle cost, scales to the spike
(a) many-small	persistent warm cluster	low latency via warm cache

Why this works — concept by concept:

• Serverless for independent spikes — no cluster to idle between bursts, autoscaled per batch, billed per DCU-second — ideal when jobs don't share state.
• Warm cluster for many-small — a persistent, autoscaling cluster keeps executors and cache hot, so latency-sensitive repeat jobs avoid cold start.
• Workload shape decides — the choice is driven by burstiness, shared state, and latency needs, not by "serverless is always better."
• Shared data layer — because both hit gs:// and one Metastore, routing by shape costs nothing in data plumbing.
• Cost — Serverless cost is O(work per batch) with zero idle; a warm cluster trades a small standing cost for O(1) cold-start latency on many jobs.

PySpark
Topic — pyspark
Batch PySpark job-design problems

Practice →

Optimization Topic — optimization Runtime-selection and cost-tradeoff problems

Practice →

---

Cheat sheet — Dataproc recipes

Ephemeral cluster with idle TTL.

gcloud dataproc clusters create eph \
  --region=us-central1 --num-workers=2 --max-idle=30m

Primary + Spot secondary workers.

gcloud dataproc clusters create mixed \
  --region=us-central1 \
  --num-workers=4 \
  --num-secondary-workers=20 --secondary-worker-type=spot

Submit a PySpark job to a cluster.

gcloud dataproc jobs submit pyspark gs://b/code/job.py \
  --cluster=mixed --region=us-central1 -- gs://b/in/ gs://b/out/

Read GCS, write BigQuery (PySpark).

df = spark.read.parquet("gs://lake/events/")
(df.write.format("bigquery")
   .option("table", "analytics.daily")
   .option("temporaryGcsBucket", "lake/tmp")
   .mode("overwrite").save())

Attach a shared Dataproc Metastore.

gcloud dataproc clusters create eph \
  --region=us-central1 --max-idle=30m \
  --dataproc-metastore=projects/PROJ/locations/us-central1/services/shared-hms

Submit a Serverless batch.

gcloud dataproc batches submit pyspark gs://b/code/job.py \
  --region=us-central1 \
  --properties=spark.dynamicAllocation.maxExecutors=20

Runtime picker.

Situation	Runtime
Independent, bursty batch job	Dataproc Serverless
Many tiny latency-sensitive jobs sharing cache	Persistent autoscaling cluster
Lift-and-shift Hadoop/Hive with HDFS scratch	Ephemeral cluster + GCS
Cheap elastic compute you can lose	Spot secondary workers + EFM

---

Frequently asked questions

What is Google Cloud Dataproc?

Dataproc is Google Cloud's managed Hadoop and Spark service. It provisions clusters running stock open-source engines — Spark, PySpark, Spark SQL, Hive, Trino, Flink — in about ninety seconds, exposes cluster and job APIs, and lets you delete the cluster when the job is done. Because your data lives in Google Cloud Storage rather than on the cluster, a Dataproc cluster is a disposable, per-job compute lease rather than a standing system you operate all year.

Dataproc vs Dataproc Serverless — when do I use each?

A Dataproc cluster is provisioned infrastructure: you pick machine types and worker counts (or attach an autoscaling policy) and you can keep it warm for many jobs. Dataproc Serverless for Spark removes the cluster entirely — you gcloud dataproc batches submit a workload and Google autoscales a runtime per batch, billing by DCU-second. Use Serverless for independent, bursty batch jobs where you don't want to size anything; use a persistent cluster for many small latency-sensitive jobs that benefit from warm executors and cached data, or when you need long-running components and fine machine control.

What are primary vs secondary workers?

Primary workers run both a YARN NodeManager (compute) and an HDFS DataNode (storage), so they hold durable cluster data and are standard, non-preemptible VMs. Secondary workers run a NodeManager only — no HDFS — so they add pure compute capacity and are the correct place for cheap Spot (preemptible) VMs. The rule is: put HDFS and the Spark application master on primary workers, and scale cheap, reclaimable compute with secondary Spot workers.

Should I store data in HDFS on Dataproc?

Only transiently. Cluster-local HDFS is fast scratch for shuffle spill and in-job intermediates, but it dies with the cluster and partly rides on reclaimable Spot workers, so it must never be your source of truth. Keep inputs and outputs in Google Cloud Storage (gs://) via the GCS connector; that is what makes deleting the cluster harmless and lets many clusters share the same data.

How do preemptible workers save money without losing jobs?

Spot (preemptible) VMs cost a fraction of on-demand but can be reclaimed with about thirty seconds' notice, so you place them only on secondary workers, which store no HDFS. YARN reschedules tasks from a reclaimed node onto survivors, so losing Spot capacity slows a job instead of failing it. To make aggressive Spot ratios safe, keep a real floor of primary workers and enable Enhanced Flexibility Mode so shuffle data lives on primaries and is never lost when a Spot node disappears.

Dataproc vs Dataflow vs BigQuery?

BigQuery is a serverless SQL warehouse — use it when the work is SQL over tables. Dataflow runs Apache Beam pipelines on a fully auto-tuned, clusterless runner — use it for greenfield streaming or batch written in Beam. Dataproc runs your existing Spark/Hadoop/Hive code on managed clusters (or Serverless batches) — use it to lift Spark-shaped workloads to GCP with minimal change, often reading from and writing to BigQuery through the connector. They coexist: Dataproc for Spark, BigQuery for SQL analytics, Dataflow for Beam.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Dataproc idea above, from the primary/secondary worker split to the ephemeral autoscaling cluster and the GCS-to-BigQuery connector job, maps to a hands-on practice room where you write the Spark against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you keep this cluster cheap and reliable under preemption?" holds up under a senior interviewer's depth probes.

Practice Spark SQL problems now →
PySpark drills →