Hadoop MapReduce作业卡死，因为auxService:mapreduce_shuffle不存在 [英] Hadoop MapReduce Job stuck because auxService:mapreduce_shuffle does not exist

问题描述

我检查了多个具有相同问题的帖子，并且解决方案始终是将以下内容添加到yarn-site.xml

I've checked multiple posts with the same questions, and the solution is always adding the following to the yarn-site.xml

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarm.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

我介绍了配置中的两个属性并重新启动了yarn.问题仍然存在.

I covered both properties in the config and restarted yarn. The problem still remains.

错误是:

17/02/15 15:43:34 INFO mapreduce.Job: Task Id : attempt_1487202110321_0001_m_000000_2, Status : FAILED
Container launch failed for container_1487202110321_0001_01_000007 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
    at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:375)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

我原本希望输入错误，但似乎找不到.尝试将stackoverflow上的内容直接复制到xml文件中，仍然无效.

I was hoping for a typo but can't seem to find it. Tried directly copy what's on stackoverflow into the xml file, still doesn't work.

我还能尝试什么?

由于错误提示aux_service应该为auxService，因此我根据修改了yarn-site.xml，将所有aux-service更改为auxService，但仍然无法正常工作.

Since the error says the aux_service should be auxService, i modified the yarn-site.xml according, changing all aux-service to auxService, but it's still not working.

万一有兴趣的人，我会打电话给这个命令

In case anyone's interested, I call this command

hadoop jar hadoop-streaming-2.7.1.jar \
       -input /user/myfolder/input1/* \
       -output /user/myfolder/output1 \
       -mapper <path>/<to>/<mapper>/mapper.py \
       -reducer <path>/<to>/<reducer>/reducer.py

当我已经在/usr/local/cellar/hadoop/2.7.1/libexec/share/hadoop/tools/lib/

我是笨蛋.校对脚本的人！

I'm a dumbass. proof-read the script guys!

推荐答案

将yarn-site.xml中的属性名称更新为yarn.nodemanager.aux-services，

Update the property name in yarn-site.xml as yarn.nodemanager.aux-services,

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

这篇关于Hadoop MapReduce作业卡死，因为auxService:mapreduce_shuffle不存在的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！