DEV Community

Cover image for How to Use EMR APIs?
Preecha
Preecha

Posted on

How to Use EMR APIs?

TL;DR

AWS EMR (Elastic MapReduce) APIs manage big data clusters running Hadoop, Spark, Hive, and Presto. You create clusters, submit jobs as steps, auto-scale based on workload, and terminate when done. Authentication uses AWS IAM. Use Apidog to validate cluster configurations, test job submissions against the API structure, and document your data pipelines.

Try Apidog today

Introduction

AWS EMR is AWS's managed Hadoop and Spark service for analytics, machine learning, and ETL workloads. It processes data on EC2 instances without requiring you to operate the underlying Hadoop infrastructure yourself.

An EMR cluster is made up of EC2 instances. When creating one, define:

  • Instance types for primary, core, and task nodes
  • Applications such as Spark, Hadoop, Hive, Presto, or HBase
  • Bootstrap actions for setup scripts
  • Steps that run your jobs

The EMR API lets you automate the full cluster lifecycle: create clusters, submit jobs, monitor execution, configure scaling, and terminate resources when processing is complete.

💡 If you’re building data pipelines, Apidog can help you test cluster configurations, validate job definitions, and document EMR workflows before running expensive data processing jobs.

Test AWS APIs with Apidog - free

By the end of this guide, you’ll be able to:

  • Create and configure EMR clusters through the API and AWS CLI
  • Submit Spark and Hive jobs as EMR steps
  • Configure auto-scaling policies
  • Monitor cluster and step status
  • Reduce costs with transient clusters, instance fleets, and Spot instances

Authenticate with AWS

EMR uses standard AWS IAM authentication. For application code, prefer an AWS SDK. For command-line workflows, use the AWS CLI. Both handle AWS Signature Version 4 (SigV4) signing for you.

Use the AWS SDK

Install the EMR SDK package:

npm install @aws-sdk/client-emr
Enter fullscreen mode Exit fullscreen mode

Create an EMR client using environment variables for credentials:

import { EMRClient, RunJobFlowCommand } from '@aws-sdk/client-emr'

const client = new EMRClient({
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY
  }
})
Enter fullscreen mode Exit fullscreen mode

Avoid hard-coding credentials in source code. In production, use IAM roles where possible.

Use the AWS CLI

The AWS CLI signs requests automatically when your credentials are configured:

aws emr list-clusters --region us-east-1
Enter fullscreen mode Exit fullscreen mode

Direct API requests and SigV4

The EMR API requires AWS Signature Version 4. If you call the API directly, you must sign every request. In practice, use an AWS SDK, boto3, or the AWS CLI unless you specifically need to implement SigV4 yourself.

IAM permissions

A basic policy for EMR management can include the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:*",
        "ec2:Describe*",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Use this as a starting point, then scope actions and resources down for production workloads.

Create an EMR cluster

Create a basic Spark cluster with the AWS CLI

The following command creates a three-node cluster with Spark and Hadoop installed:

aws emr create-cluster \
  --name "My Spark Cluster" \
  --release-label emr-7.0.0 \
  --applications Name=Spark Name=Hadoop \
  --instance-type m5.xlarge \
  --instance-count 3 \
  --service-role EMR_DefaultRole \
  --job-flow-role EMR_EC2_DefaultRole
Enter fullscreen mode Exit fullscreen mode

Before running it, make sure that:

  1. The EMR service role exists.
  2. The EC2 instance profile role exists.
  3. Your account has enough EC2 capacity and service quota for the selected instance type.
  4. Your region supports the EMR release label and instance type.

Create a cluster with RunJobFlow

Use the RunJobFlow API when you need programmatic cluster provisioning:

{
  "Name": "Data Processing Cluster",
  "ReleaseLabel": "emr-7.0.0",
  "Applications": [
    { "Name": "Spark" },
    { "Name": "Hadoop" },
    { "Name": "Hive" }
  ],
  "Instances": {
    "MasterInstanceType": "m5.xlarge",
    "SlaveInstanceType": "m5.xlarge",
    "InstanceCount": 3,
    "KeepJobFlowAliveWhenNoSteps": true,
    "TerminationProtected": false
  },
  "Steps": [],
  "ServiceRole": "EMR_DefaultRole",
  "JobFlowRole": "EMR_EC2_DefaultRole",
  "LogUri": "s3://my-bucket/emr-logs/",
  "Tags": [
    { "Key": "Environment", "Value": "Production" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

A successful request returns a cluster ID:

{
  "JobFlowId": "j-ABC123DEF456"
}
Enter fullscreen mode Exit fullscreen mode

Store this ID. You need it to add steps, inspect status, configure scaling, and terminate the cluster.

Choose instance groups or instance fleets

EMR supports two provisioning approaches:

  • Instance groups: Use a fixed instance type for each group: primary, core, and task.
  • Instance fleets: Provide multiple instance types and capacity targets. EMR selects available capacity based on your configuration.

Use instance fleets when capacity availability or Spot pricing is a concern.

{
  "Instances": {
    "InstanceFleets": [
      {
        "Name": "MasterFleet",
        "InstanceFleetType": "MASTER",
        "TargetOnDemandCapacity": 1,
        "InstanceTypeConfigs": [
          {
            "InstanceType": "m5.xlarge"
          },
          {
            "InstanceType": "m4.xlarge"
          }
        ]
      },
      {
        "Name": "CoreFleet",
        "InstanceFleetType": "CORE",
        "TargetOnDemandCapacity": 2,
        "TargetSpotCapacity": 4,
        "InstanceTypeConfigs": [
          {
            "InstanceType": "m5.2xlarge"
          },
          {
            "InstanceType": "m4.2xlarge"
          }
        ],
        "LaunchSpecifications": {
          "SpotSpecification": {
            "TimeoutDurationMinutes": 60,
            "TimeoutAction": "SWITCH_TO_ON_DEMAND"
          }
        }
      }
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Submit jobs as EMR steps

EMR runs jobs as steps. Steps execute sequentially by default, so define their failure behavior carefully.

Add a Spark step with the AWS CLI

Use command-runner.jar to invoke spark-submit:

aws emr add-steps \
  --cluster-id j-ABC123DEF456 \
  --steps '[
    {
      "Name": "Process Data",
      "ActionOnFailure": "CONTINUE",
      "HadoopJarStep": {
        "Jar": "command-runner.jar",
        "Args": [
          "spark-submit",
          "--deploy-mode",
          "cluster",
          "--class",
          "com.example.DataProcessor",
          "s3://my-bucket/jars/processor.jar",
          "s3://my-bucket/input/",
          "s3://my-bucket/output/"
        ]
      }
    }
  ]'
Enter fullscreen mode Exit fullscreen mode

Use --deploy-mode cluster when the Spark driver should run on the EMR cluster rather than on the machine submitting the command.

Add steps with AddJobFlowSteps

The equivalent API request looks like this:

{
  "JobFlowId": "j-ABC123DEF456",
  "Steps": [
    {
      "Name": "Spark ETL Job",
      "ActionOnFailure": "CONTINUE",
      "HadoopJarStep": {
        "Jar": "command-runner.jar",
        "Args": [
          "spark-submit",
          "--executor-memory",
          "4g",
          "--executor-cores",
          "2",
          "s3://my-bucket/scripts/process.py",
          "--input",
          "s3://my-bucket/input/",
          "--output",
          "s3://my-bucket/output/"
        ]
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Set ActionOnFailure

Choose the failure behavior that matches your pipeline:

Value Behavior
TERMINATE_CLUSTER Stops the cluster when the step fails.
CANCEL_AND_WAIT Cancels remaining steps and keeps the cluster running.
CONTINUE Continues to the next step after a failure.

For transient ETL clusters, TERMINATE_CLUSTER can prevent idle compute costs after a critical failure. For debugging workflows, CANCEL_AND_WAIT can be useful because the cluster remains available for inspection.

Add a Hive step

Use hive-script through command-runner.jar:

{
  "Name": "Hive Query",
  "HadoopJarStep": {
    "Jar": "command-runner.jar",
    "Args": [
      "hive-script",
      "--run-hive-script",
      "--args",
      "-f",
      "s3://my-bucket/scripts/transform.q"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

Configure auto-scaling

EMR can add or remove nodes based on CloudWatch metrics. Auto-scaling is useful when workload volume changes over time.

Create an auto-scaling policy

This policy adds two nodes when available memory is low:

aws emr put-auto-scaling-policy \
  --cluster-id j-ABC123DEF456 \
  --instance-group-id ig-ABC123 \
  --auto-scaling-policy '{
    "Constraints": {
      "MinCapacity": 2,
      "MaxCapacity": 10
    },
    "Rules": [
      {
        "Name": "ScaleOut",
        "Description": "Add nodes when memory is high",
        "Action": {
          "SimpleScalingPolicyConfiguration": {
            "AdjustmentType": "CHANGE_IN_CAPACITY",
            "ScalingAdjustment": 2,
            "CoolDown": 300
          }
        },
        "Trigger": {
          "CloudWatchAlarmDefinition": {
            "ComparisonOperator": "GREATER_THAN",
            "EvaluationPeriods": 3,
            "MetricName": "MemoryAvailableMB",
            "Namespace": "AWS/ElasticMapReduce",
            "Period": 300,
            "Threshold": 2000,
            "Statistic": "AVERAGE"
          }
        }
      }
    ]
  }'
Enter fullscreen mode Exit fullscreen mode

Validate the metric logic before applying it. For example, MemoryAvailableMB represents free memory, so scale-out thresholds should be selected carefully for the behavior you intend.

Metrics to use for scaling

Common EMR metrics include:

Metric Use
MemoryAvailableMB Available memory across the cluster.
MemoryTotalMB Total cluster memory.
HDFSUtilization Percentage of HDFS storage in use.
AppsRunning Number of running YARN applications.
AppsPending Number of YARN applications waiting to run.

Monitor clusters and jobs

List running clusters 

aws emr list-clusters --states RUNNING
Enter fullscreen mode Exit fullscreen mode

Describe a cluster 

aws emr describe-cluster --cluster-id j-ABC123DEF456
Enter fullscreen mode Exit fullscreen mode

The response includes cluster state, installed applications, logging location, and the primary node DNS name:

{
  "Cluster": {
    "Id": "j-ABC123DEF456",
    "Name": "My Cluster",
    "Status": {
      "State": "RUNNING",
      "StateChangeReason": {},
      "Timeline": {
        "CreationDateTime": "2026-03-24T10:00:00.000Z"
      }
    },
    "Applications": [
      { "Name": "Spark", "Version": "3.5.0" }
    ],
    "InstanceCollectionType": "INSTANCE_GROUP",
    "LogUri": "s3://my-bucket/emr-logs/",
    "MasterPublicDnsName": "ec2-12-34-56-78.compute-1.amazonaws.com"
  }
}
Enter fullscreen mode Exit fullscreen mode

List cluster steps 

aws emr list-steps --cluster-id j-ABC123DEF456
Enter fullscreen mode Exit fullscreen mode

A completed step has a status similar to this:

{
  "Id": "s-ABC123",
  "Name": "Process Data",
  "Status": {
    "State": "COMPLETED",
    "Timeline": {
      "StartDateTime": "2026-03-24T10:05:00.000Z",
      "EndDateTime": "2026-03-24T11:30:00.000Z"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Monitor through CloudWatch

EMR publishes CloudWatch metrics including:

  • JobsFailed
  • JobsRunning
  • MemoryAvailableMB
  • MemoryTotalMB
  • HDFSUtilization

Use these metrics for dashboards, alarms, and auto-scaling triggers.

Optimize EMR costs

Use Spot instances for task capacity

Task nodes are a good candidate for Spot instances because they process data but do not store HDFS data.

{
  "Name": "TaskGroup",
  "InstanceRole": "TASK",
  "InstanceType": "m5.2xlarge",
  "InstanceCount": 4,
  "Market": "SPOT",
  "BidPrice": "0.10"
}
Enter fullscreen mode Exit fullscreen mode

If a Spot task node is interrupted, work can continue on the remaining nodes.

Use transient clusters

For scheduled batch jobs, create a cluster, run the required steps, and terminate it after completion:

{
  "KeepJobFlowAliveWhenNoSteps": false,
  "Steps": [
    { "...": "step 1" },
    { "...": "step 2" }
  ]
}
Enter fullscreen mode Exit fullscreen mode

With KeepJobFlowAliveWhenNoSteps set to false, the cluster terminates after all steps finish.

Provide multiple instance options

Instance fleets let EMR select from several instance types:

{
  "InstanceTypeConfigs": [
    {
      "InstanceType": "m5.2xlarge",
      "BidPrice": "0.15"
    },
    {
      "InstanceType": "m4.2xlarge",
      "BidPrice": "0.12"
    },
    {
      "InstanceType": "c5.2xlarge",
      "BidPrice": "0.10"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Providing alternatives can improve capacity availability and reduce cost compared with relying on a single instance type.

Test EMR requests with Apidog

EMR clusters can be expensive to run, so validate your request payloads before using them in a production workflow.

Image

1. Validate cluster configurations

Save cluster templates in Apidog and add tests that check required fields before sending a request:

pm.test('Cluster has required applications', () => {
  const config = pm.request.body.toJSON()
  const apps = config.Applications.map(a => a.Name)

  pm.expect(apps).to.include('Spark')
})

pm.test('Instance types are valid', () => {
  const config = pm.request.body.toJSON()
  const types = ['m5.xlarge', 'm5.2xlarge', 'm4.xlarge']

  pm.expect(types).to.include(config.Instances.MasterInstanceType)
})
Enter fullscreen mode Exit fullscreen mode

2. Validate step definitions

Test that a Spark step uses the expected command and arguments:

pm.test('Spark step has valid args', () => {
  const step = pm.request.body.toJSON().Steps[0]
  const args = step.HadoopJarStep.Args

  pm.expect(args[0]).to.eql('spark-submit')
  pm.expect(args).to.include('--deploy-mode')
})
Enter fullscreen mode Exit fullscreen mode

3. Store environment variables

Keep environment-specific values out of request bodies:

AWS_REGION: us-east-1
EMR_SERVICE_ROLE: EMR_DefaultRole
EMR_EC2_ROLE: EMR_EC2_DefaultRole
S3_LOG_BUCKET: my-emr-logs
S3_SCRIPTS_BUCKET: my-emr-scripts
Enter fullscreen mode Exit fullscreen mode

Test AWS APIs with Apidog - free

Common errors and fixes

ValidationError: ServiceRole is not valid

Cause: The IAM service role does not exist or is not configured for EMR.

Fix: Create the service role in IAM or use the AWS default role:

EMR_DefaultRole_V2
Enter fullscreen mode Exit fullscreen mode

Failed to provision EC2 instances

Cause: The selected instance type is unavailable in the Availability Zone, or your account has reached a service limit.

Fix:

  • Use instance fleets with multiple instance types.
  • Request a service limit increase.
  • Select different instance types.

Step failed with application exit code 1

Cause: The Spark or Hadoop application failed.

Fix:

  1. Check the EMR step status.
  2. Open the S3 location configured by LogUri.
  3. Review stderr and stdout for the failed step.
  4. Reproduce the job with the same input paths and arguments if needed.

Cluster stuck in STARTING

Cause: Bootstrap actions failed or the cluster does not have required permissions.

Fix:

  • Check the EC2 instance console output.
  • Verify that bootstrap scripts are accessible in S3.
  • Verify that the EC2 instance profile has S3 access.

Alternatives and comparisons

Feature AWS EMR Google Dataproc Azure HDInsight Databricks
Managed Hadoop/Spark ✓ ✓ ✓ Spark only
AWS integration Excellent Limited Limited Good
Serverless option EMR Serverless Dataproc Serverless Limited ✓
Cost Spot support Preemptible VMs Spot instances Good
ML support EMR Studio Vertex AI Synapse MLflow built-in

EMR provides deep AWS integration. Databricks offers stronger Spark tooling. Dataproc can be a better fit for GCP users.

Real-world use cases

Data lake ETL

A retail company processes daily sales data. An EMR cluster reads CSV files from S3, transforms them with Spark, and writes Parquet files to a data lake. The cluster runs for two hours each day and then terminates.

Log analytics

A SaaS company processes application logs from S3. Spark aggregates metrics and writes results to a data warehouse. Auto-scaling adds task nodes when log volume increases.

Machine learning pipelines

A data science team trains models on EMR. Spark reads feature data from S3, trains models with MLlib, and exports results to SageMaker for serving.

Wrap up

You can use the EMR API and AWS CLI to:

  • Create clusters with the RunJobFlow API
  • Submit Spark or Hive work as steps
  • Configure auto-scaling policies
  • Monitor execution with EMR APIs and CloudWatch
  • Control costs with Spot instances and transient clusters

A practical next-step checklist:

  1. Set up IAM roles for EMR.
  2. Create a small test cluster.
  3. Submit a simple Spark job.
  4. Review logs in S3.
  5. Add scaling and cost controls after validating the workload.

Test AWS APIs with Apidog - free

FAQ

What’s the difference between primary, core, and task nodes?

  • Primary: Runs cluster management services such as YARN ResourceManager and HDFS NameNode.
  • Core: Runs data processing and stores HDFS data.
  • Task: Runs data processing only and does not store HDFS data. This makes task nodes suitable for Spot instances.

How do I SSH into the primary node? 

aws emr ssh --cluster-id j-ABC123DEF456 --key-pair-file my-key.pem
Enter fullscreen mode Exit fullscreen mode

Can I run Jupyter notebooks on EMR?

Yes. Use EMR Studio, enable the JupyterHub application, or use EMR Notebooks.

What is EMR Serverless?

EMR Serverless is a serverless option for submitting Spark and Hive jobs without managing clusters. You pay per job run, which can be useful for sporadic workloads.

How do I read from DynamoDB?

Use the DynamoDB connector:

spark-submit --conf spark.hadoop.dynamodb.servicename=dynamodb \
  --conf spark.hadoop.dynamodb.input.tableName=MyTable \
  --conf spark.hadoop.dynamodb.output.tableName=MyTable \
  --conf spark.hadoop.dynamodb.region=us-east-1 \
  my-job.jar
Enter fullscreen mode Exit fullscreen mode

What release label should I use?

Use the latest stable release label that is compatible with your applications, such as emr-7.x for Spark 3.x. Keep versions consistent across environments and check application compatibility in the release notes.

How do I troubleshoot failed steps?

  • Check the step status: 
  aws emr describe-step
Enter fullscreen mode Exit fullscreen mode
  • Review S3 logs: 
  s3://your-log-bucket/logs/j-ABC123/steps/s-DEF123/
Enter fullscreen mode Exit fullscreen mode
  • SSH to the primary node and inspect logs: 
  /mnt/var/log/
Enter fullscreen mode Exit fullscreen mode

Top comments (0)