AWS EMR S3DistCp:auxService:mapreduce_shuffle不存在 [英] AWS EMR S3DistCp: The auxService:mapreduce_shuffle does not exist
问题描述
我已通过SSH连接到AWS EMR v5.4.0实例,我想调用s3distcp.此链接演示了如何设置emr步骤来调用它,但是当我运行它时,出现以下错误:
I am connected to an AWS EMR v5.4.0 instance over SSH and I want to call s3distcp. This link demonstrates how to setup an emr step to call it, but when I run it I get the following error:
Container launch failed for container_1492469375740_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:390)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
我遵循了此处的说明,但仍然没有效果.
I followed the instructions here but it still didn't work.
推荐答案
It turns out I needed to restart the yarn nodemanager service after configuring mapreduce_shuffle
:
$ initctl list | grep yarn
hadoop-yarn-resourcemanager start/running, process 1256
hadoop-yarn-proxyserver start/running, process 702
hadoop-yarn-nodemanager start/running, process 896
$ sudo stop hadoop-yarn-nodemanager
$ sudo start hadoop-yarn-nodemanager
另外,如果它有助于将 yarn-site.xml
文件放在以下位置:/etc/hadoop/conf/yarn-site.xml
.它已经有 yarn.nodemanager.aux-services
的条目,但未配置mapreduce_shuffle:
Also, in case it helps the yarn-site.xml
file was located at: /etc/hadoop/conf/yarn-site.xml
. It already had an entry for yarn.nodemanager.aux-services
but mapreduce_shuffle wasn't configured:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
所以我这样添加它:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
这篇关于AWS EMR S3DistCp:auxService:mapreduce_shuffle不存在的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!