amazon emr is the managed way to run Apache Spark, Hive, Trino, Presto, Flink, and HBase on AWS without owning the operational nightmare underneath them — you rent a running framework instead of building one from a fleet of raw EC2 boxes. It provisions and configures the cluster, installs a curated, version-pinned stack, wires it to S3 through EMRFS, and hands you back a spark-submit endpoint or a SQL prompt. What used to be a week of Hadoop plumbing is now an API call that returns a cluster in minutes, or — with the serverless variant — no cluster at all.
That convenience hides a set of decisions that separate an engineer who has actually run EMR in production from one who has only read the console. Which nodes hold state and which are disposable? Where is it safe to put Spot capacity, and what happens in the two minutes before an interruption? When do you drop the cluster entirely and go serverless? And how do you get ACID transactions, time travel, and schema evolution out of files sitting in an S3 bucket? This guide walks the four ideas an interviewer will actually probe — the primary / core / task topology and instance fleets, Spot-fleet economics and interruption handling, the EMR Serverless application / job-run model, and Apache Iceberg on S3 with the Glue Data Catalog — and pairs each with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.
When you want hands-on reps immediately after reading, drill the distributed-query patterns on the Spark SQL practice library →, rehearse the data-layout decisions on the partitioning practice set →, and sharpen the cost-and-performance calls on the optimization practice set →.
On this page
- Why Amazon EMR still matters in 2026
- EMR on EC2 cluster topology
- Spot fleets & cost optimization
- EMR Serverless — application & job-run
- Iceberg & the lakehouse on EMR
- Cheat sheet — Amazon EMR recipes
- Frequently asked questions
- Practice on PipeCode
1. Why Amazon EMR still matters in 2026
EMR is managed big-data frameworks with storage decoupled from compute — that one fact decides everything downstream
The one-sentence invariant: EMR runs open-source big-data engines for you on AWS, with data living in S3 rather than on the cluster, so compute becomes something you spin up, use, and throw away. Every design choice — transient clusters, Spot task nodes, serverless job-runs, Iceberg tables — follows from decoupling storage from compute. The cluster is cattle; the data in S3 is the pet.
What EMR actually gives you.
-
A curated framework stack. Each EMR release (e.g. the
emr-7.xline) pins compatible versions of Spark, Hive, Trino/Presto, Flink, HBase, Hudi, Iceberg, and Delta together, so you are not resolving Hadoop-vs-Spark version conflicts by hand. -
Provisioning and configuration. EMR launches the EC2 (or EKS, or serverless) capacity, installs the stack, applies your
configurationsoverrides, and exposes YARN, Spark History Server, and application UIs. - EMRFS — the S3 connector. EMR reads and writes S3 through EMRFS, an implementation of the Hadoop filesystem that adds S3-specific handling (consistent listings, encryption, IAM role integration). Your tables live in S3, not HDFS.
- Catalog integration. EMR can use the AWS Glue Data Catalog as its Hive metastore, so table definitions survive the cluster and are shared with Athena, Redshift Spectrum, and other engines.
The three deployment models — the fork every design starts from.
- EMR on EC2. The classic model: you get a real cluster of EC2 instances with a primary, core, and task groups, full control over the OS, bootstrap actions, and long-running or transient lifecycles. Maximum control, maximum ops surface.
- EMR on EKS. EMR runs Spark as pods on a Kubernetes cluster you already operate. You reuse EKS capacity and tooling; EMR supplies the optimized Spark runtime and job submission. Best when a platform team has standardized on Kubernetes.
- EMR Serverless. No cluster, no instances to size. You create an application and submit job-runs; AWS provisions and scales workers automatically and bills per second of vCPU/memory used. Best for spiky, intermittent, or bursty workloads.
Where EMR sits against the alternatives.
- vs self-managed Hadoop/Spark on EC2. Rolling your own means you patch the OS, resolve version conflicts, and build autoscaling and Spot handling yourself. EMR does all of that and adds S3 optimizations you would otherwise reinvent.
- vs AWS Glue ETL. Glue is serverless Spark with a job abstraction and a tighter, more opinionated surface; EMR gives you the full framework catalog (Trino, Flink, HBase, notebooks) and knob-level control. Reach for EMR when you outgrow Glue's abstraction or need engines Glue does not offer.
- vs Databricks / Snowflake. Those are managed platforms with their own runtimes and billing; EMR keeps you on open-source engines and open table formats you can lift to any other environment, at the cost of more assembly.
What interviewers listen for.
- Do you say "storage is decoupled — data lives in S3, the cluster is disposable" early? — senior signal.
- Can you name the three deployment models and when each wins without prompting? — required framing.
- Do you place task nodes on Spot and keep state off them rather than blanket-Spot the whole cluster? — the single most common depth probe.
- Do you reach for open table formats (Iceberg/Hudi/Delta) plus the Glue Catalog for the lakehouse, not raw Parquet directories? — modern signal.
Worked example — a transient cluster that outlives nothing but the data
Detailed explanation. The canonical EMR pattern is a transient cluster: it launches, runs one or more steps, writes results to S3, and terminates itself. Because the data is in S3 and the table metadata is in Glue, nothing of value dies with the cluster — you pay only for the minutes the job ran. The same job could run on a cluster that lived for months; the transient version just makes the "compute is disposable" idea concrete.
Question. Launch a cluster that runs a single Spark step reading from and writing to S3, then shuts down on its own. Show what survives after termination.
Input.
| resource | lives where | survives cluster termination? |
|---|---|---|
| input Parquet | s3://lake/raw/ |
yes |
| Spark job jar | s3://artifacts/etl.jar |
yes |
| output table |
s3://lake/curated/ + Glue Catalog |
yes |
| YARN / HDFS scratch | on the cluster | no |
Code.
aws emr create-cluster \
--name "nightly-etl" \
--release-label emr-7.2.0 \
--applications Name=Spark \
--use-default-roles \
--instance-groups \
InstanceGroupType=MASTER,InstanceType=m5.xlarge,InstanceCount=1 \
InstanceGroupType=CORE,InstanceType=m5.xlarge,InstanceCount=2 \
--steps '[{"Type":"Spark","Name":"etl","ActionOnFailure":"TERMINATE_CLUSTER",
"Args":["--deploy-mode","cluster","s3://artifacts/etl.jar"]}]' \
--auto-terminate
Step-by-step explanation. --release-label pins the framework stack; --applications Name=Spark installs only Spark. EMR provisions one MASTER (primary) and two CORE instances, runs the step in cluster deploy mode so the driver lives on the cluster, and — because of --auto-terminate — shuts the whole cluster down the moment the step queue drains. The job read s3://lake/raw/, wrote s3://lake/curated/, and registered the table in Glue, so all three outputs persist while the compute evaporates.
Output.
| after termination | state |
|---|---|
| cluster | terminated, billing stopped |
s3://lake/curated/ |
new Parquet written |
| Glue Catalog table | present, queryable by Athena |
| cost | ~minutes of 3 instances, nothing idle |
Rule of thumb. If the answer to "what do we lose if this cluster dies right now?" is anything other than "in-flight compute," your data is on the wrong layer — push it to S3 and let the cluster be disposable.
2. EMR on EC2 cluster topology
Primary, core, and task nodes divide a cluster into one coordinator, a stateful HDFS tier, and a disposable compute tier
An EMR on EC2 cluster is not a uniform pool of machines — it is three node types with sharply different jobs, and an interviewer who asks "walk me through an EMR cluster" wants all three in order with the state boundary called out. Get this wrong and you will put Spot capacity where it corrupts data.
The three node types.
-
Primary node (exactly one, sometimes three for HA). Runs the coordinators: the YARN ResourceManager, the HDFS NameNode, and cluster-management daemons. In
clientdeploy mode the Spark driver also lives here. Lose the primary and the cluster is gone — so it is never Spot in a job you care about. - Core nodes (one or more). Run the YARN NodeManager (they execute tasks) and the HDFS DataNode (they store HDFS blocks). Core nodes are stateful: losing one can lose HDFS block replicas and shuffle data, so removing them is risky and EMR only scales them down carefully.
- Task nodes (zero or more). Run the YARN NodeManager only — pure compute, no HDFS. They hold nothing durable, so they can appear and disappear freely. This is exactly why task nodes are the correct home for Spot capacity.
Instance groups vs instance fleets — the two ways to specify capacity.
- Instance groups. Each group is a single instance type with a target count (and optional Spot). Simple, but a single Spot instance type means a single Spot pool — thin diversification.
- Instance fleets. Each fleet (there is one fleet per role) can mix up to a wide list of instance types, expressed in capacity units, split across On-Demand and Spot targets, with an allocation strategy. Fleets are the modern, resilient choice because they diversify Spot across many pools.
Managed scaling — let EMR size the compute tier.
- What it watches. EMR-managed scaling monitors YARN metrics (pending containers/memory, allocated vs available) and adds or removes core/task capacity to keep the cluster right-sized against the workload.
-
The bounds you set. You give a
MinimumCapacityUnits,MaximumCapacityUnits, an optionalMaximumOnDemandCapacityUnits, andMaximumCoreCapacityUnits. EMR scales task capacity aggressively and core capacity conservatively, respecting those ceilings. - Why the split matters. Because core nodes hold HDFS, managed scaling keeps a stable core floor and does most of its up/down movement on task nodes — the disposable tier.
Worked example — sizing a fleet cluster with core on-demand and task on spot
Detailed explanation. The standard resilient layout is a small On-Demand core fleet to hold HDFS and shuffle safely, plus a large Spot task fleet that carries the bulk of the compute. You express capacity in units so EMR can satisfy a target from any mix of the listed instance types.
Question. Define a cluster with one On-Demand primary, an On-Demand core fleet of 64 units, and a task fleet targeting 256 Spot units drawn from several instance types. Which role carries HDFS, and where does interruption risk sit?
Input. Three fleets: primary (1 unit On-Demand), core (64 units On-Demand), task (256 units Spot across m5.2xlarge, m5a.2xlarge, m6g.2xlarge, r5.2xlarge).
Code.
[
{"InstanceFleetType": "MASTER",
"TargetOnDemandCapacity": 1,
"InstanceTypeConfigs": [{"InstanceType": "m5.xlarge", "WeightedCapacity": 1}]},
{"InstanceFleetType": "CORE",
"TargetOnDemandCapacity": 64,
"InstanceTypeConfigs": [
{"InstanceType": "m5.2xlarge", "WeightedCapacity": 8},
{"InstanceType": "m5a.2xlarge", "WeightedCapacity": 8}]},
{"InstanceFleetType": "TASK",
"TargetSpotCapacity": 256,
"LaunchSpecifications": {"SpotSpecification":
{"TimeoutDurationMinutes": 20, "TimeoutAction": "SWITCH_TO_ON_DEMAND",
"AllocationStrategy": "capacity-optimized"}},
"InstanceTypeConfigs": [
{"InstanceType": "m5.2xlarge", "WeightedCapacity": 8},
{"InstanceType": "m5a.2xlarge", "WeightedCapacity": 8},
{"InstanceType": "m6g.2xlarge", "WeightedCapacity": 8},
{"InstanceType": "r5.2xlarge", "WeightedCapacity": 8}]}
]
Step-by-step explanation. Each WeightedCapacity says how many capacity units one instance contributes, so EMR reaches 64 core units with 8 of the 8-unit instances, and 256 task units with 32 of them — from whichever of the four Spot pools has capacity. The MASTER and CORE fleets use TargetOnDemandCapacity, so the coordinator and HDFS tier never run on Spot. The TASK fleet uses TargetSpotCapacity with capacity-optimized allocation and a timeout that falls back to On-Demand if Spot cannot be filled in 20 minutes.
Output.
| fleet | capacity | purchasing | holds HDFS? | interruptible? |
|---|---|---|---|---|
| MASTER | 1 unit | On-Demand | NameNode | no |
| CORE | 64 units | On-Demand | yes (DataNode) | no |
| TASK | 256 units | Spot | no | yes |
Rule of thumb. Put exactly enough capacity On-Demand to hold HDFS and shuffle safely (primary + a modest core fleet) and let the big, elastic, interruptible task fleet ride Spot.
Amazon EMR interview question on node roles
Question. An interviewer says: "Your nightly Spark job keeps failing at the reduce stage with FetchFailedException after the cluster scales down. The cluster is all Spot, including core nodes. Diagnose it and fix the topology." Walk through it.
Solution Using an On-Demand core floor with Spot only on task nodes
Code.
{
"CORE": {"TargetOnDemandCapacity": 32, "TargetSpotCapacity": 0,
"note": "stateful: HDFS + shuffle blocks live here"},
"TASK": {"TargetOnDemandCapacity": 0, "TargetSpotCapacity": 512,
"AllocationStrategy": "capacity-optimized",
"note": "compute only: safe to lose"}
}
Step-by-step trace.
| symptom | root cause | fix |
|---|---|---|
FetchFailedException at shuffle read |
Spot core node reclaimed mid-job | move core to On-Demand |
| lost shuffle blocks | shuffle data sat on interrupted core/task | keep a stable On-Demand core floor |
| cascading stage retries | executors chasing gone blocks | diversify + capacity-optimized Spot on task |
-
FetchFailedExceptionmeans a reducer could not fetch shuffle output from a node that vanished — a classic symptom of an interrupted node that held shuffle blocks. - Because core nodes were on Spot, an interruption reclaimed a DataNode/NodeManager holding both HDFS replicas and shuffle data mid-stage, so the fetch had nowhere to read from.
- The fix is topological: pin core to On-Demand so the stateful tier is never reclaimed, and put Spot only on task nodes, which hold no shuffle-critical state EMR cannot rebuild.
- Diversifying the task fleet across instance types with
capacity-optimizedallocation further lowers the chance of a correlated interruption wave.
Output:
| topology | shuffle stability | Spot savings |
|---|---|---|
| all-Spot (before) | fragile — fetch failures | maximal but unreliable |
| On-Demand core + Spot task (after) | stable | most of the savings, reliably |
Why this works — concept by concept:
- State boundary — the fix hinges on knowing core nodes are stateful (HDFS + shuffle) and task nodes are not; you protect state and expose only stateless compute to interruption.
- FetchFailedException — a shuffle-fetch failure is the tell that a node holding intermediate data disappeared; the cure is keeping that data on non-interruptible capacity.
- On-Demand floor — a modest On-Demand core fleet guarantees a durable home for HDFS and shuffle so a stage can always complete.
- Diversified Spot task — spreading task capacity across many instance pools makes a broad simultaneous reclaim unlikely, so the elastic tier stays mostly full.
- Cost — you trade a small always-on On-Demand core cost for reliability, while the dominant compute cost still runs on Spot at roughly 60–90% off On-Demand.
Spark SQL
Topic — spark-sql
Distributed Spark SQL and shuffle-stage problems
3. Spot fleets & cost optimization
Spot task fleets cut compute cost 60–90% — the price is interruption, and the whole game is draining gracefully instead of failing hard
Spot instances are spare EC2 capacity sold at a deep discount that AWS can reclaim with a two-minute warning. On EMR they are the single biggest cost lever, and the entire discipline is arranging the cluster so that a reclaim is a shrink, not a crash. Say the rule in one breath: Spot goes on task nodes, diversified across many instance types, with capacity-optimized allocation, and EMR decommissions gracefully on the two-minute notice.
Where Spot is safe.
- Task nodes — yes. They hold no HDFS and no durable state; losing one just removes executors, and EMR (plus managed scaling) replaces the capacity.
- Core nodes — with caution. Core-on-Spot is possible but risky because they hold HDFS blocks and shuffle data; most production clusters keep a stable On-Demand core floor.
- Primary — never (for real work). One primary; losing it kills the cluster.
Allocation strategies for the Spot fleet.
-
capacity-optimized. EMR provisions Spot from the pools with the most spare capacity right now, which minimizes the probability of interruption. This is the default recommendation for long-running stages. -
price-capacity-optimized. Balances the lowest price against capacity depth — usually the best all-round choice, cheap and stable. -
lowest-price(legacydiversified). Chases the cheapest pools; higher interruption risk. Rarely the right call for a job that must finish. -
Diversify the instance list. List many families and sizes (
m5,m5a,m6g,r5,c5, multiple sizes) so a shortage in one pool does not starve the fleet.
Handling the interruption.
- Two-minute notice. AWS emits a rebalance recommendation and then a termination notice; EMR reacts by starting a graceful YARN decommission of the node so running containers finish or are rescheduled and no new work lands on it.
-
Timeout + fallback. On the fleet you set
TimeoutDurationMinutesandTimeoutAction=SWITCH_TO_ON_DEMAND, so if Spot cannot be provisioned, EMR falls back to On-Demand rather than hanging. - Checkpoint long stages. For very long Spark jobs, checkpointing (or Iceberg's incremental commits) bounds how much work a reclaim can cost you.
Other cost levers beyond Spot.
-
Transient clusters. Terminate on completion (
--auto-terminate); pay only for the run. -
Right-size and cap managed scaling. Set
MaximumOnDemandCapacityUnitsso a Spot shortfall cannot silently balloon your On-Demand bill. -
Graviton (
m6g/r6g). ARM instances typically deliver better price/performance for Spark; add them to the fleet list. - S3 over HDFS. Keep durable data in S3 so you never pay to keep a cluster alive just to hold data.
Worked example — a diversified spot task fleet with on-demand fallback
Detailed explanation. The resilient Spot configuration lists many instance types, sets capacity-optimized (or price-capacity-optimized) allocation, and provides an On-Demand fallback so the job never stalls waiting for spare capacity that is not there.
Question. Configure a task fleet that targets 400 Spot units across five instance types, uses price-capacity-optimized allocation, and falls back to On-Demand after 15 minutes if Spot cannot fill.
Input. Task fleet target = 400 units; instance types m5.2xlarge, m5a.2xlarge, m6g.2xlarge, c5.2xlarge, r5.2xlarge, each weighted 8 units.
Code.
{
"InstanceFleetType": "TASK",
"TargetSpotCapacity": 400,
"LaunchSpecifications": {
"SpotSpecification": {
"AllocationStrategy": "price-capacity-optimized",
"TimeoutDurationMinutes": 15,
"TimeoutAction": "SWITCH_TO_ON_DEMAND"
}
},
"InstanceTypeConfigs": [
{"InstanceType": "m5.2xlarge", "WeightedCapacity": 8, "BidPriceAsPercentageOfOnDemandPrice": 100},
{"InstanceType": "m5a.2xlarge", "WeightedCapacity": 8, "BidPriceAsPercentageOfOnDemandPrice": 100},
{"InstanceType": "m6g.2xlarge", "WeightedCapacity": 8, "BidPriceAsPercentageOfOnDemandPrice": 100},
{"InstanceType": "c5.2xlarge", "WeightedCapacity": 8, "BidPriceAsPercentageOfOnDemandPrice": 100},
{"InstanceType": "r5.2xlarge", "WeightedCapacity": 8, "BidPriceAsPercentageOfOnDemandPrice": 100}
]
}
Step-by-step explanation. EMR needs 50 instances (400 ÷ 8) and draws them from whichever of the five pools is cheapest-yet-deep under price-capacity-optimized. BidPriceAsPercentageOfOnDemandPrice: 100 caps the Spot bid at the On-Demand price so you never overpay. If, after 15 minutes, Spot cannot supply the full 400 units, SWITCH_TO_ON_DEMAND fills the remainder with On-Demand so the job proceeds. When a pool is reclaimed, EMR decommissions those nodes gracefully and re-provisions from the remaining pools.
Output.
| condition | fleet behaviour |
|---|---|
| Spot plentiful | 400 units from cheapest/deepest pools, ~70% off |
| one pool reclaimed | graceful drain, refill from other 4 pools |
| Spot scarce > 15 min | remainder filled On-Demand, job continues |
Rule of thumb. Diversify wide, allocate price-capacity-optimized, and always set an On-Demand fallback — a job that finishes slightly more expensively beats a job that hangs waiting for Spot.
Amazon EMR interview question on Spot interruptions
Question. Your Spark job on a Spot task fleet loses 30% of its executors when a Spot pool is reclaimed mid-stage. The job recovers but reprocesses a lot of work. What is EMR doing during those two minutes, and what two changes reduce the reprocessing cost?
Solution Using graceful decommission plus diversification and checkpointing
Code.
spark.conf.set("spark.decommission.enabled", "true") # graceful drain, not a hard kill
spark.conf.set("spark.storage.decommission.enabled", "true") # migrate shuffle/cache off draining nodes
df = spark.read.parquet("s3://lake/events/")
df.repartition(2000).write.mode("append").parquet("s3://lake/curated/")
spark.sparkContext.setCheckpointDir("s3://lake/_checkpoints/") # bound lineage; recover from S3, not stage 0
Step-by-step trace.
| t | event | EMR / Spark action |
|---|---|---|
| t0 | Spot reclaim signalled | rebalance + 2-min termination notice received |
| t0..t0+2m | node draining | YARN graceful decommission: no new containers, running tasks finish/migrate |
| t0+2m | node terminated | lost executors' in-progress tasks rescheduled elsewhere |
| after | stage recovery | tasks recompute only lost partitions, reading inputs from S3 |
- On the two-minute notice EMR stops scheduling new containers on the doomed node and lets in-flight tasks finish or migrate — that is graceful decommission, not a hard kill.
- With
spark.storage.decommission.enabled, shuffle and cached blocks are migrated off the draining node, so reducers do not later hitFetchFailedException. - Diversifying the task fleet means a single pool reclaim removes a smaller fraction of executors, shrinking the recompute set.
- Checkpointing (or reading from Iceberg snapshots) caps lineage so recovery re-reads durable S3 inputs instead of recomputing a long chain from the beginning.
Output:
| change | effect on reprocessing |
|---|---|
| decommission block migration on | reducers avoid refetch failures |
| wider Spot diversification | fewer executors lost per reclaim |
| checkpoint long lineage | recompute bounded to lost partitions |
Why this works — concept by concept:
- Two-minute contract — Spot's guaranteed warning is what makes graceful drain possible; EMR uses the window to quiesce the node instead of losing work abruptly.
-
Decommission block migration — moving shuffle/cache off a draining node converts a hard
FetchFailedExceptioninto a soft, already-relocated read. - Diversification — spreading across pools decorrelates interruptions so no single reclaim takes a large slice of the fleet.
- Checkpoint / snapshot boundary — a durable S3 checkpoint bounds recompute to the lost partitions rather than the whole lineage.
- Cost — recovery work drops to O(lost partitions) instead of O(full stage), so the Spot discount is kept without paying it back in reprocessing.
Optimization
Topic — optimization
Cost and performance optimization problems
4. EMR Serverless — application & job-run
EMR Serverless removes the cluster entirely — you size an application and submit job-runs, and AWS scales workers per second
EMR Serverless is the deployment model for teams who want EMR's engines without provisioning or babysitting a cluster. There is no primary, no core, no task nodes to reason about. Instead you create an application bound to one framework (Spark or Hive) and a release, then submit job-runs against it; AWS allocates workers on demand, scales them with the job, and bills per second of vCPU, memory, and storage consumed. Say it in one breath: application = the configured runtime, job-run = one submission, workers = auto-scaled units you never launch by hand.
The object model.
-
Application. A named, versioned runtime for one engine type (
type: SPARKortype: HIVE) on a chosenreleaseLabel. It holds defaults: worker sizing, network config, pre-initialized capacity, and auto-stop settings. It is not a running cluster — it is a template that spawns capacity when jobs arrive. -
Job-run. One unit of work submitted to an application: a
spark-submit-style entry point (jar/py) with arguments, or a Hive query, plus its own resource overrides. Job-runs are independent and can run concurrently against the same application. -
Workers. The compute units EMR Serverless provisions per job. You specify per-worker
cpu,memory, anddisk; EMR decides how many to run and scales them up as the job's parallelism demands and down as it finishes.
Pre-initialized capacity — the cold-start dial.
- On-demand (default). Workers are provisioned when a job-run starts. Cheapest at rest (you pay nothing idle) but each job pays a cold-start delay while workers spin up.
- Pre-initialized capacity. You keep a pool of warm workers (e.g. 5 drivers + 50 executors of a given size) ready, so job-runs start in seconds. You pay for the warm pool while it is initialized, so it suits latency-sensitive or frequent jobs.
- Auto-stop. The application idles down after a configurable timeout, so a pre-init pool does not bill forever after the last job.
Sizing and scaling.
-
Worker sizing.
cpuin vCPU (1–many),memoryin GB within allowed ratios,diskfor shuffle/spill. Match the memory:core ratio to the workload (wide shuffles want more memory). - Automatic scaling. EMR Serverless adds workers as Spark requests executors and releases them when stages complete — you do not write a scaling policy.
-
maximumCapacity. A ceiling on total vCPU/memory/disk the application may consume, so a runaway job cannot blow the budget.
When Serverless wins — and when it does not.
- Wins for: spiky, intermittent, or unpredictable workloads; teams that do not want to manage clusters; many small independent jobs; getting to "just run my Spark" fast.
- Loses for: steady 24/7 high-utilization workloads (a right-sized long-running cluster on Spot is often cheaper); jobs needing engines beyond Spark/Hive (Trino, Flink, HBase → EMR on EC2/EKS); deep OS-level customization via bootstrap actions.
Worked example — one application, a job-run, and warm capacity
Detailed explanation. The everyday Serverless pattern is: create a Spark application (optionally with pre-init capacity), then fire job-runs at it with start-job-run. The application persists between jobs; each job-run is billed only for the workers and seconds it used.
Question. Create a Spark EMR Serverless application with a small pre-initialized pool, then submit one Spark job that reads and writes S3. What is billed, and what makes the job start fast?
Input. Application type=SPARK, releaseLabel=emr-7.2.0, pre-init = 1 driver (4 vCPU/16 GB) + 10 executors (4 vCPU/16 GB); one job-run entry point s3://artifacts/job.py.
Code.
## create the application with warm pre-initialized capacity
aws emr-serverless create-application \
--type SPARK --release-label emr-7.2.0 --name analytics \
--initial-capacity '{
"DRIVER": {"workerCount": 1, "workerConfiguration": {"cpu": "4vCPU", "memory": "16GB"}},
"EXECUTOR": {"workerCount": 10, "workerConfiguration": {"cpu": "4vCPU", "memory": "16GB"}}}' \
--maximum-capacity '{"cpu": "400vCPU", "memory": "1600GB", "disk": "8000GB"}' \
--auto-stop-configuration '{"enabled": true, "idleTimeoutMinutes": 15}'
## submit a job-run against it
aws emr-serverless start-job-run \
--application-id 00fabc... \
--execution-role-arn arn:aws:iam::111122223333:role/EMRServerlessRole \
--job-driver '{"sparkSubmit": {"entryPoint": "s3://artifacts/job.py"}}'
Step-by-step explanation. create-application defines the runtime and pre-provisions 1 driver + 10 executors that sit warm, so the first job-run skips cold start and begins in seconds. maximum-capacity caps total consumption so no job exceeds 400 vCPU. start-job-run submits the Spark script; EMR Serverless uses the warm workers first, then auto-scales more executors if the job needs them, and releases them when stages finish. auto-stop shuts the warm pool after 15 idle minutes so you stop paying for it.
Output.
| item | billed |
|---|---|
| pre-init warm pool | per-second while initialized (fast starts) |
| job-run executors | per-second vCPU/memory/disk actually used |
| auto-scaled executors | added during the job, released after |
| idle after 15 min | nothing — application auto-stopped |
Rule of thumb. Turn on pre-initialized capacity only when start latency matters or jobs are frequent; for occasional batch, run pure on-demand and accept the cold start to pay nothing at rest.
Amazon EMR interview question on choosing Serverless vs a cluster
Question. You run ~40 short, unpredictable Spark jobs a day, each 3–8 minutes, spread across business hours, and users complain about start latency. Would you use EMR Serverless or a long-running EMR on EC2 cluster, and how would you configure it to be both fast and cheap?
Solution Using EMR Serverless with pre-initialized capacity and auto-stop
Code.
{
"type": "SPARK",
"releaseLabel": "emr-7.2.0",
"initialCapacity": {
"DRIVER": {"workerCount": 2, "workerConfiguration": {"cpu": "4vCPU", "memory": "16GB"}},
"EXECUTOR": {"workerCount": 20, "workerConfiguration": {"cpu": "4vCPU", "memory": "16GB"}}
},
"maximumCapacity": {"cpu": "400vCPU", "memory": "1600GB"},
"autoStopConfiguration": {"enabled": true, "idleTimeoutMinutes": 10}
}
Step-by-step trace.
| requirement | Serverless feature | outcome |
|---|---|---|
| bursty, unpredictable arrival | on-demand + auto-scaling | capacity appears per job, no idle cluster |
| complaint: slow starts | pre-initialized capacity (warm pool) | job-runs begin in seconds, not minutes |
| cheap at rest | auto-stop after 10 idle min | warm pool released outside busy windows |
| budget guard | maximumCapacity ceiling | no single burst blows the bill |
- Short, spiky, independent jobs are the textbook Serverless case: a long-running cluster would sit mostly idle between the 3–8-minute jobs, burning money.
- The start-latency complaint is solved by pre-initialized capacity — a modest warm pool (2 drivers + 20 executors) so job-runs skip cold start.
- Auto-stop at 10 idle minutes releases the warm pool during quiet stretches, so you pay for warmth only around actual demand.
-
maximumCapacitybounds total consumption so a bad job or a burst cannot run away with cost.
Output:
| dimension | long-running cluster | EMR Serverless (chosen) |
|---|---|---|
| idle cost | high (cluster always on) | near-zero (auto-stop) |
| start latency | fast (already up) | fast (pre-init warm pool) |
| ops burden | scaling, patching, Spot handling | none |
Why this works — concept by concept:
- Application vs job-run — the application holds the warm, configured runtime while each job-run is billed independently, matching spiky demand without an idle cluster.
- Pre-initialized capacity — a warm worker pool trades a little standing cost for seconds-not-minutes starts, directly fixing the latency complaint.
- Automatic scaling — workers grow and shrink per job, so you never over- or under-provision a fixed cluster.
- Auto-stop — releasing the warm pool when idle is what keeps the fast-start design cheap outside business hours.
- Cost — total spend is O(seconds of actual work + a small warm-pool window) instead of O(24h cluster), which is why Serverless beats a cluster for intermittent load.
Spark SQL
Topic — spark-sql
Spark job and query-tuning problems
5. Iceberg & the lakehouse on EMR
Apache Iceberg turns S3 files into an ACID table — EMR plus Iceberg plus the Glue Catalog is the modern lakehouse
The last piece is how EMR gets warehouse behaviour — transactions, schema evolution, time travel — out of plain files in S3. The answer is an open table format: Iceberg (the most common on EMR), Hudi, or Delta Lake. EMR ships all three, but Iceberg is the reference choice, and the standard stack is Spark on EMR writing Iceberg tables whose metadata pointer lives in the Glue Data Catalog and whose data and metadata files live in S3.
What a table format adds over a directory of Parquet.
- ACID commits. A write produces a new immutable snapshot and atomically swaps the catalog's metadata pointer, so readers always see a consistent table — no half-written partitions, no eventual-consistency surprises.
-
Snapshots & time travel. Every commit is a snapshot; you can query
TIMESTAMP AS OF/VERSION AS OFto read the table as it was, and roll back a bad write. - Schema & partition evolution. Add, drop, or rename columns, and even change the partition spec, without rewriting the data — old files keep their layout, new files use the new one.
-
Hidden partitioning. Iceberg derives partition values from a transform (
days(ts),bucket(16, id)) so queries filter on the raw column and Iceberg prunes partitions automatically — noWHERE dt='...'gymnastics. -
MERGE INTO/ row-level ops. Upserts, deletes, and CDC land as first-class SQL against the S3 table.
The catalog — where the pointer lives.
-
Glue Data Catalog as Iceberg catalog. EMR configures
spark.sql.catalog.<name>=org.apache.iceberg.spark.SparkCatalogwithcatalog-impl=org.apache.iceberg.aws.glue.GlueCatalog, so table metadata is registered in Glue and shared with Athena, Redshift Spectrum, and other engines. -
EMRFS + S3 for storage. The Iceberg
warehousepath is an S3 prefix; data files (Parquet), manifest files, manifest lists, andmetadata.jsonall live there, read/written through EMRFS withio-impl=org.apache.iceberg.aws.s3.S3FileIO.
Operational must-dos.
-
Compaction. Streaming/upsert workloads create many small files; run
rewrite_data_filesto compact them or query performance degrades. -
Expire snapshots. Old snapshots retain old data files;
expire_snapshotsreclaims S3 storage — balance against how far back you need time travel. - Commit retries. Concurrent writers may collide on the pointer swap; Iceberg retries the commit, so design writers to be idempotent.
Wiring it up on EMR — steps and bootstrap actions.
-
Steps (
add-steps/ step API). A step is a unit of work submitted to a running cluster — aspark-submit, a Hive script — queued and run in order. Use steps to run the actual Iceberg jobs. -
Bootstrap actions. Scripts that run on every node at launch, before applications start — install a library, drop a config file, set an OS tunable. Use them for node-level setup; use EMR
configurations(classification JSON) for framework settings like the Iceberg catalog.
Worked example — create and upsert an Iceberg table on S3 via Glue
Detailed explanation. The end-to-end Iceberg-on-EMR flow is: configure the Spark session with a Glue-backed Iceberg catalog pointing at an S3 warehouse, CREATE TABLE with a partition transform, then MERGE INTO to upsert. The table is ACID, time-travellable, and visible to Athena — all from Spark SQL.
Question. On EMR, create an Iceberg table db.orders partitioned by day of updated_at, stored in S3 and catalogued in Glue, then upsert a batch by order_id. Show the config and the SQL.
Input. Warehouse s3://lake/warehouse/, Glue catalog name glue, upsert batch of 2 rows (one update, one insert).
Code.
spark = (SparkSession.builder
.config("spark.sql.catalog.glue", "org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.glue.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
.config("spark.sql.catalog.glue.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
.config("spark.sql.catalog.glue.warehouse", "s3://lake/warehouse/")
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.getOrCreate())
spark.sql("""
CREATE TABLE IF NOT EXISTS glue.db.orders (
order_id BIGINT, amount DOUBLE, updated_at TIMESTAMP)
USING iceberg
PARTITIONED BY (days(updated_at))
""")
spark.sql("""
MERGE INTO glue.db.orders t
USING updates s ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
""")
Step-by-step explanation. The spark.sql.catalog.glue.* configs register an Iceberg catalog named glue whose implementation is GlueCatalog (metadata pointer in Glue) and whose storage is S3FileIO under the s3://lake/warehouse/ prefix. CREATE TABLE ... USING iceberg PARTITIONED BY (days(updated_at)) uses a hidden-partition transform, so writers and readers reference updated_at directly. MERGE INTO matches on order_id, updating existing rows and inserting new ones, and commits a new snapshot by atomically advancing the Glue metadata pointer.
Output.
| what happened | where it landed |
|---|---|
| new data + delete files | s3://lake/warehouse/db/orders/data/ |
new metadata.json + manifests |
s3://lake/warehouse/db/orders/metadata/ |
| current pointer advanced | Glue Data Catalog (db.orders) |
| new snapshot | queryable now; old snapshot still time-travellable |
Rule of thumb. Point the Iceberg catalog at Glue and the warehouse at S3, partition with a transform not a raw column, and you get an ACID, evolvable, time-travelling table that every AWS query engine can read.
Amazon EMR interview question on the lakehouse stack
Question. A streaming EMR job upserts into an Iceberg table every minute. After a week, read queries are slow and S3 storage keeps growing even though row count is flat. Diagnose it and give the maintenance you would schedule.
Solution Using compaction and snapshot expiration
Code.
-- 1) compact the many tiny files each minute-ly commit produced
CALL glue.system.rewrite_data_files(
table => 'db.orders',
options => map('target-file-size-bytes','536870912') -- 512 MB targets
);
-- 2) reclaim storage held by old snapshots (keep 3 days of time travel)
CALL glue.system.expire_snapshots(
table => 'db.orders',
older_than => TIMESTAMP '2026-09-12 00:00:00',
retain_last => 10
);
Step-by-step trace.
| symptom | root cause | maintenance |
|---|---|---|
| slow reads | thousands of tiny files per partition |
rewrite_data_files → 512 MB files |
| growing S3, flat rows | every upsert kept old data + delete files |
expire_snapshots reclaims them |
| planning overhead | huge manifest lists to scan | compaction also rewrites manifests |
- Minute-ly upserts each write a new snapshot with small data/delete files, so after a week each partition holds thousands of tiny files — read planning and I/O explode.
-
rewrite_data_filescompacts those into ~512 MB files, collapsing delete files into the data and cutting the file count read queries must open. - Even after compaction, old snapshots still reference the old files, so S3 keeps growing;
expire_snapshotsdrops snapshots past the retention window and deletes their now-unreferenced files. - Scheduling both (e.g. hourly compaction, daily expiration) keeps read latency and storage flat for a streaming upsert table.
Output:
| metric | before | after maintenance |
|---|---|---|
| files per partition | thousands (tiny) | tens (~512 MB) |
| S3 storage | growing weekly | bounded to retention window |
| read latency | degraded | restored |
Why this works — concept by concept:
- Small-file problem — frequent commits fragment the table; compaction restores large, scan-efficient files that dominate read cost.
- Snapshot retention — Iceberg keeps every snapshot's files for time travel, so storage only shrinks when you explicitly expire old snapshots.
- rewrite_data_files — the maintenance procedure that merges data and delete files into right-sized objects and rewrites manifests.
- expire_snapshots — the procedure that reclaims S3 by dropping out-of-retention snapshots and their orphaned files.
- Cost — read cost falls toward O(rows / file-size) instead of O(number of tiny files), and storage becomes O(live data + retention window) rather than unbounded.
Partitioning
Topic — partitioning
Partition-transform and file-layout problems
Cheat sheet — Amazon EMR recipes
Spot-heavy instance-fleet cluster (CLI).
aws emr create-cluster --name spot-etl --release-label emr-7.2.0 \
--applications Name=Spark \
--instance-fleets file://fleets.json --use-default-roles --auto-terminate
Managed scaling policy.
{"ComputeLimits": {"UnitType": "InstanceFleetUnits",
"MinimumCapacityUnits": 64, "MaximumCapacityUnits": 512,
"MaximumOnDemandCapacityUnits": 96, "MaximumCoreCapacityUnits": 96}}
Spark + Iceberg + Glue session config.
--conf spark.sql.catalog.glue=org.apache.iceberg.spark.SparkCatalog
--conf spark.sql.catalog.glue.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
--conf spark.sql.catalog.glue.warehouse=s3://lake/warehouse/
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
EMR Serverless start-job-run.
aws emr-serverless start-job-run --application-id 00fabc... \
--execution-role-arn arn:aws:iam::111122223333:role/EMRServerlessRole \
--job-driver '{"sparkSubmit": {"entryPoint": "s3://artifacts/job.py"}}'
Add a step to a running cluster.
aws emr add-steps --cluster-id j-XXXX \
--steps Type=Spark,Name=load,Args=[s3://artifacts/etl.jar]
Bootstrap action stub (runs on every node at launch).
#!/bin/bash
sudo pip3 install great_expectations # node-level setup before apps start
Deployment-model picker.
| Situation | Model |
|---|---|
| Full control, long-running or many engines | EMR on EC2 |
| Already standardized on Kubernetes | EMR on EKS |
| Spiky / intermittent, no cluster ops wanted | EMR Serverless |
| Big elastic compute, cost-sensitive | EMR on EC2 with Spot task fleet |
Frequently asked questions
What is Amazon EMR?
Amazon EMR is a managed AWS service for running open-source big-data frameworks — Apache Spark, Hive, Trino/Presto, Flink, HBase, and more — without building and operating the cluster yourself. EMR provisions and configures the compute, installs a version-pinned framework stack, connects it to Amazon S3 through EMRFS, and integrates with the Glue Data Catalog. You run it as a real cluster (EMR on EC2), on Kubernetes (EMR on EKS), or with no cluster at all (EMR Serverless).
What is the difference between EMR on EC2, EMR on EKS, and EMR Serverless?
EMR on EC2 gives you a real cluster with primary, core, and task nodes and full control over the OS, bootstrap actions, and lifecycle — the most powerful and the most operational. EMR on EKS runs EMR's Spark runtime as pods on a Kubernetes cluster you already operate, so you reuse EKS capacity and tooling. EMR Serverless removes the cluster entirely: you create an application and submit job-runs, and AWS auto-scales workers and bills per second — ideal for spiky, intermittent workloads.
Which EMR nodes are safe to run on Spot?
Task nodes are the safe home for Spot because they run compute only and hold no HDFS data, so losing one just removes executors that EMR replaces. Core nodes hold HDFS blocks and shuffle data, so running them on Spot risks FetchFailedException and lost data when a node is reclaimed — keep a stable On-Demand core floor. The single primary node should never be Spot, because losing it destroys the whole cluster.
What are EMR instance fleets?
Instance fleets are the modern way to specify EMR on EC2 capacity: one fleet per role (primary, core, task), each able to mix many instance types expressed in weighted capacity units and split across On-Demand and Spot targets. A fleet uses an allocation strategy such as capacity-optimized or price-capacity-optimized to diversify Spot across many pools, which dramatically lowers interruption risk compared with a single-instance-type instance group.
How does EMR use Apache Iceberg and the Glue Data Catalog?
EMR ships Iceberg (and Hudi and Delta) and configures Spark with an Iceberg catalog whose implementation is GlueCatalog, so table metadata is registered in the AWS Glue Data Catalog while data files, manifests, and metadata.json live in S3 via S3FileIO. This gives you ACID commits, snapshots and time travel, schema and partition evolution, and MERGE INTO upserts over S3 — and the Glue-catalogued table is instantly readable by Athena, Redshift Spectrum, and other engines.
Does EMR store data in HDFS or S3?
Both exist, but S3 is where durable data belongs. EMR clusters have HDFS on their core nodes for transient, cluster-local data (shuffle spill, intermediate results), but that dies with the cluster. Your tables and datasets should live in S3, read and written through EMRFS, so compute stays disposable — you can terminate the cluster, or run transient and serverless jobs, without losing anything of value.
Practice on PipeCode
Pipecode.ai is Leetcode for Data Engineering — every EMR idea above, from the primary/core/task state boundary to Spot-fleet diversification, EMR Serverless sizing, and Iceberg compaction on S3, maps to a hands-on practice room where you build the query and the layout against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "where do you put Spot and why?" holds up under a senior interviewer's depth probes.
Top comments (0)