YARN的工作历史无法访问 [英] YARN job history not accessible

问题描述

我正在使用从源代码构建的最新hadoop版本3.0.0.我已经启动并运行了时间轴服务，并配置了hadoop以便将其用于作业历史记录.但是，当我在resoucemanager UI中单击历史记录时，出现以下错误:-

I am using the latest hadoop version 3.0.0 build from source code. I have my timeline service up and running and have configured hadoop to use that for job history also. But when I click on history in the resoucemanager UI I get the below error:-

HTTP ERROR 404

Problem accessing /jobhistory/job/job_1444395439959_0001. Reason:

    NOT_FOUND

有人可以指出我在这里想念的东西吗?以下是我的yarn-site.xml:-

Can someone please point out what I am missing here. Following is my yarn-site.xml:-

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <description>The hostname of the Timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>0.0.0.0</value>
</property>
<property>
  <description>Address for the Timeline server to start the RPC server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10200</value>
</property>

<property>
  <description>The http address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>

<property>
  <description>The https address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:8190</value>
</property>

<property>
  <description>Handler thread count to serve the client RPC requests.</description>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>10</value>
</property>
<property>
  <description>Indicate to ResourceManager as well as clients whether
  history-service is enabled or not. If enabled, ResourceManager starts
  recording historical data that Timelien service can consume. Similarly,
  clients can redirect to the history service when applications
  finish if this is enabled.</description>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Store class name for history store, defaulting to file system
  store</description>
  <name>yarn.timeline-service.generic-application-history.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>
<property>
     <description>URI pointing to the location of the FileSystem path where the history will be persisted.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.uri</name>
     <value>/tmp/yarn/system/history</value>
</property>
<property>
     <description>T-file compression types used to compress history data.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.compression-type</name>
     <value>none</value>
</property>



 <property>
     <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>

和我的mapred-site.xml

and my mapred-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>mapreduce.job.emit-timeline-data</name>
<value>true</value>
</property>
</configuration>

JPS输出:

6022 NameNode
27976 NodeManager
27859 ResourceManager
6139 DataNode
6310 SecondaryNameNode
28482 ApplicationHistoryServer
29230 Jps

推荐答案

如果要通过YARN RM Web UI查看日志，则需要启用日志聚合.为此，您需要在yarn-site.xml中设置以下参数:

If you want to see the logs through YARN RM web UI, then you need to enable the log aggregation. For that, you need to set the following parameters, in yarn-site.xml:

  <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
  </property>
  <property>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
  </property>
  <property>
      <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
      <value>logs</value>
  </property>

如果未启用日志聚合，则NM将在本地存储日志.通过上述设置，日志将在HDFS中的"/app-logs/{用户名}/logs/"处进行汇总.在此文件夹下，您可以找到到目前为止运行的所有应用程序的日志.再次，日志保留由配置参数"yarn.log-aggregation.retain-seconds"(保留聚合日志的时间)确定.

If you do not enable log aggregation, then NMs will store the logs locally. With the above settings, the logs are aggregated in HDFS at "/app-logs/{username}/logs/". Under this folder, you can find logs for all the applications run so far. Again the log retention is determined by the configuration parameter "yarn.log-aggregation.retain-seconds" (how long to retain the aggregated logs).

当MapReduce应用程序正在运行时，您可以从YARN的Web UI访问日志.应用程序完成后，将通过Job History Server提供日志.

When the MapReduce applications are running, then you can access the logs from the YARN's web UI. Once the application is completed, the logs are served through Job History Server.

此外，在yarn-site.xml中设置以下配置参数:

Also, set following configuration parameter in yarn-site.xml:

<property>
  <name>yarn.log.server.url</name>
  <value>http://{job-history-hostname}:8188/jobhistory/logs</value>
</property>

这篇关于YARN的工作历史无法访问的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！