Differences Introduced in 4.x

Before Amazon EMR release 4.0.0, Amazon EMR software was referenced by its AMI versions. With Amazon EMR release 4.0.0 and later, releases are now referenced by their release label.

The following are ways of specifying release:

Console

Previously, in Version you chose the AMI Version and still do for 2.x and 3.x releases.

Choose EMR release for 4.x or later releases.

CLI

For AMI version releases 2.x and 3.x, specify --ami-version 3.x.x.

For EMR releases emr-4.0.0 or later, use --release-label emr-4.x.x.

API and SDK

In the API you provide either AmiVersion or ReleaseLabel depending on the respective releases.

In the Java SDK, the following RunJobFlowRequest call specifies an AMI version:

RunJobFlowRequest request = new RunJobFlowRequest()
			.withName("AmiVersion Cluster")
			.withAmiVersion("3.8.0")
			.withInstances(new JobFlowInstancesConfig()
				.withEc2KeyName("myKeyPair")
				.withInstanceCount(1)
				.withKeepJobFlowAliveWhenNoSteps(true)
				.withMasterInstanceType("m3.xlarge")
				.withSlaveInstanceType("m3.xlarge");

The following RunJobFlowRequest call uses a release label instead:

RunJobFlowRequest request = new RunJobFlowRequest()
			.withName("ReleaseLabel Cluster")
			.withReleaseLabel("emr-4.0.0")
			.withInstances(new JobFlowInstancesConfig()
				.withEc2KeyName("myKeyPair")
				.withInstanceCount(1)
				.withKeepJobFlowAliveWhenNoSteps(true)
				.withMasterInstanceType("m3.xlarge")
				.withSlaveInstanceType("m3.xlarge");

In AMI versions 2.x and 3.x, applications were installed in any number of ways including: the NewSupportedProducts parameter for the RunJobFlow action, using bootstrap actions, and the Step action. With Amazon EMR release 4.x, there is a new, simpler way to install applications on your cluster:

Console

On the Quick Create page, applications are grouped in bundles. In emr-4.0.0 you can choose from All Applications (Hadoop, Spark, Pig, Hive, and Mahout), Core Hadoop (Hadoop, Hive, and Pig), and Spark (Spark and Hadoop-YARN). On the Advanced cluster configuration page, you can select the exact applications you want to install, and optionally edit the default configuration for each application. For more information about editing application configuration, see Configuring Applications.

CLI

Installing applications is not changed using the CLI although you no longer provide application configuration with the Args parameter. You instead use the Configurations parameter to provide the path to a JSON-formatted file containing a set of configuration objects. You can store the file locally or in Amazon S3. For more information, see Configuring Applications.

Java SDK

With emr-4.0.0, the preferred way to install applications using Java is to supply a list of Applications to RunJobFlowRequest. Using the AWS SDK for Java, this looks like:

List<Application> myApps = new ArrayList<Application>();
myApps.add("Spark","Hive","Mahout");

RunJobFlowRequest request = new RunJobFlowRequest()
	.withName("My EMR Cluster")
	.withReleaseLabel("emr-4.0.0")
	.withApplications(myApps)
	.withInstances(new JobFlowInstancesConfig()
    	.withEc2KeyName("myKeyName")
    	.withInstanceCount(1)
    	.withKeepJobFlowAliveWhenNoSteps(true)
    	.withMasterInstanceType("m3.xlarge")
    	.withSlaveInstanceType("m3.xlarge");
);

Application configuration is simplified with emr-4.0.0. Every application that you are able to specify in the Applications parameter supplied to the RunJobFlow action can be configured using a Configuration object. Furthermore, native applications are no longer configured by bootstrap actions. For example, this method replaces the configure-hadoop and configure-daemons bootstrap actions, which were used to configure certain applications. Those bootstrap actions are replaced by configurations. Configuration objects consist of a classification, properties, and optional nested configurations. A classification refers to an application-specific configuration file. Properties are the settings you want to change in that file. You typically supply configurations in a list, allowing you to edit multiple configuration files in one JSON object.

Instead of using the s3get bootstrap action to copy objects to each node, use a custom bootstrap action which will run AWS CLI on each node. The syntax would look like:

aws s3 cp s3://mybucket/myfolder/myobject myFolder/

In the AWS SDK for Java, you do this with a ScriptBootstrapActionConfig:

ScriptBootstrapActionConfig s3Config = new ScriptBootstrapActionConfig()
	.withPath("file:///usr/bin/aws")
	.withArgs("s3", "cp","s3://mybucket/myfolder/myobject","myFolder/");

With the AWS CLI, you can launch the cluster with the bootstrap action using the following command:

aws emr create-cluster --release-label emr-4.0.0 --instance-type m3.xlarge --instance-count 1 --bootstrap-actions Path=file:///usr/bin/aws,Name="copyToAll",Args="s3","cp","s3://mybucket/myfolder/myobject","myFolder/" --use-default-roles

For more information, see Configuring Applications

In Amazon EMR AMI versions 2.x and 3.x, there was a hadoop-user-env.sh script which was not part of standard Hadoop and was used along with the configure-daemons bootstrap action to configure Hadoop environment. The script did the following:

#!/bin/bash 
export HADOOP_USER_CLASSPATH_FIRST=true; 
echo "HADOOP_CLASSPATH=/path/to/my.jar" >> /home/hadoop/conf/hadoop-user-env.sh

In Amazon EMR release 4.0.0 and later, you can do the same now with the hadoop-env configurations:

[ 
      { 
         "Classification":"hadoop-env",
         "Properties":{ 

         },
         "Configurations":[ 
            { 
               "Classification":"export",
               "Properties":{ 
                  "HADOOP_USER_CLASSPATH_FIRST":"true",
                  "HADOOP_CLASSPATH":"/path/to/my.jar"
               }
            }
         ]
      }
   ]

You may have previously used a bootstrap action configure-daemons to pass environment. For example, if you set --namenode-heap-size=2048 and --namenode-opts=-XX:GCTimeRatio=19 with configure-daemons, the equivalent JSON would look like:

[ 
      { 
         "Classification":"hadoop-env",
         "Properties":{ 

         },
         "Configurations":[ 
            { 
               "Classification":"export",
               "Properties":{ 
                  "HADOOP_DATANODE_HEAPSIZE":  "2048",
           	"HADOOP_NAMENODE_OPTS":  "-XX:GCTimeRatio=19"
               }
            }
         ]
      }
   ]

Other application environment variables are no longer defined in /home/hadoop/.bashrc. Instead, they are primarily set in /etc/default files per component or application e.g. /etc/default/hadoop. Wrapper scripts in /usr/bin/ installed by application RPMs may also set additional environment variables before involving the actual bin script.

In Amazon EMR AMI versions 2.x and 3.x, some services used custom ports. emr-4.0.0 hosts these services on open source community defaults in most cases.

In AMI versions 2.x and 3.x, the user hadoop ran all processes and owned all files. In Amazon EMR release 4.x, users exist at the application and component level. For example, here is a process status that demonstrates this user ownership model:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
hive      6452  0.2  0.7 853684 218520 ?       Sl   16:32   0:13 /usr/lib/jvm/java-openjdk/bin/java -Xmx256m -Dhive.log.dir=/var/log/hive -Dhive.log.file=hive-metastore.log -Dhive.log.threshold=INFO -Dhadoop.log.dir=/usr/lib/hadoop
hive      6557  0.2  0.6 849508 202396 ?       Sl   16:32   0:09 /usr/lib/jvm/java-openjdk/bin/java -Xmx256m -Dhive.log.dir=/var/log/hive -Dhive.log.file=hive-server2.log -Dhive.log.threshold=INFO -Dhadoop.log.dir=/usr/lib/hadoop/l
hbase     6716  0.1  1.0 1755516 336600 ?      Sl   Jun21   2:20 /usr/lib/jvm/java-openjdk/bin/java -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -Xmx1024m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/
hbase     6871  0.0  0.7 1672196 237648 ?      Sl   Jun21   0:46 /usr/lib/jvm/java-openjdk/bin/java -Dproc_thrift -XX:OnOutOfMemoryError=kill -9 %p -Xmx1024m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/
hdfs      7491  0.4  1.0 1719476 309820 ?      Sl   16:32   0:22 /usr/lib/jvm/java-openjdk/bin/java -Dproc_namenode -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop-hdfs-namenode-ip-10-71-203-213.log -Dhadoo
yarn      8524  0.1  0.6 1626164 211300 ?      Sl   16:33   0:05 /usr/lib/jvm/java-openjdk/bin/java -Dproc_proxyserver -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=yarn-yarn-
yarn      8646  1.0  1.2 1876916 385308 ?      Sl   16:33   0:46 /usr/lib/jvm/java-openjdk/bin/java -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-yarn -Dyarn.log.dir=/var/log/hadoop-yarn -Dhadoop.log.file=yarn-y
mapred    9265  0.2  0.8 1666628 260484 ?      Sl   16:33   0:12 /usr/lib/jvm/java-openjdk/bin/java -Dproc_historyserver -Xmx1000m -Dhadoop.log.dir=/usr/lib/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop

In AMI versions 2.x and 3.x, application artifacts and their configuration directories were previously installed to /home/hadoop/application. For example, if you installed Hive the directory would be /home/hadoop/hive. In EMR release 4.0.0 and later, application artifacts are installed in /usr/lib/application, so Hive would be located in /usr/lib/hive. Most configuration files are stored in /etc/application/conf, so hive-site, Hive’s configuration file, would be located at /etc/hive/conf.

Previously, log files were found in various places. In Amazon EMR 4.x and later, they are now all located under /var/log/component.

Locations for log files pushed to Amazon S3 have changed as follows:

Many scripts or programs, like /home/hadoop/contrib/streaming/hadoop-streaming.jar, are now placed on the shell login path environment so you do not need to specify the full path when executing them when using command-runner.jar. You also do not have to know the full path to command-runner.jar. command-runner.jar is also located on the AMI so there is no need to know a full URI as was the case with script-runner.jar.

The following is a list of scripts that can be executed with command-runner.jar:

The following is an example usage of command-runner.jar using the AWS CLI:

aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Name="Command Runner",Jar="command-runner.jar",Args=["spark-submit","Args..."]