YARN的工作历史未到 [英] YARN job history not coming

问题描述

我使用源代码中最新的hadoop 3.0.0版本。我的时间轴服务已启动并运行，并已配置hadoop将其用于作业历史记录。但是当我点击resoucemanager UI中的历史记录时，出现以下错误： -

  HTTP错误404 
 
问题访问/ jobhistory / job / job_1444395439959_0001。原因：
 
 NOT_FOUND 
  

有人可以指出我在这里错过了什么。以下是我的yarn-site.xml： -

 < configuration> 
 
 <！ - 网站特定的YARN配置属性 - > 
<属性> 
< description>时间轴服务Web应用程序的主机名。< / description> 
< name> yarn.timeline-service.hostname< / name> 
< value> 0.0.0.0< /值> 
< / property> 
<属性> 
< description>时间轴服务器启动RPC服务器的地址。< / description> 
< name> yarn.timeline-service.address< / name> 
< value> $ {yarn.timeline-service.hostname}：10200< /值> 
< / property> 
 
<属性> 
< description>时间轴服务Web应用程序的http地址。< / description> 
< name> yarn.timeline-service.webapp.address< / name> 
< value> $ {yarn.timeline-service.hostname}：8188< /值> 
< / property> 
 
<属性> 
< description>时间轴服务Web应用程序的https地址。< / description> 
< name> yarn.timeline-service.webapp.https.address< / name> 
< value> $ {yarn.timeline-service.hostname}：8190< /值> 
< / property> 
 
<属性> 
< description>处理程序线程数来提供客户端RPC请求。< / description> 
< name> yarn.timeline-service.handler-thread-count< / name> 
< value> 10< /值> 
< / property> 
<属性> 
< description>向ResourceManager以及客户端指示是否启用
 history-service。如果启用，ResourceManager将启动
记录Timelien服务可以消耗的历史数据。同样，如果启用了应用程序
，
客户端可以重定向到历史记录服务。< / description> 
< name> yarn.timeline-service.generic-application-history.enabled< / name> 
<值> true< /值> 
< / property> 
 
<属性> 
< description>存储历史存储的类名，默认为文件系统
 store< / description> 
< name> yarn.timeline-service.generic-application-history.store-class< / name> 
< value> org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore< / value> 
< / property> 
<属性> 
< description>指向保存历史记录的文件系统路径位置的URI< / description> 
< name> yarn.timeline-service.generic-application-history.fs-history-store.uri< / name> 
< value> / tmp / yarn / system / history< / value> 
< / property> 
<属性> 
< description>用于压缩历史数据的T文件压缩类型。< / description> 
< name> yarn.timeline-service.generic-application-history.fs-history-store.compression-type< / name> 
<值>无< /值> 
< / property> 
 
 
 
<属性> 
< name> yarn.nodemanager.aux-services< / name> 
< value> mapreduce_shuffle< /值> 
< / property> 
<属性> 
< name> yarn.nodemanager.aux-services.mapreduce_shuffle.class< / name> 
< value> org.apache.hadoop.mapred.ShuffleHandler< / value> 
< / property> 
< / configuration> 
  

和我的mapred-site.xml

 <结构> 
<属性> 
< name> mapreduce.framework.name< / name> 
<值>纱线< /值> 
< / property> 
<属性> 
<名称> mapreduce.jobhistory.address< / name> 
< value> localhost：10200< /值> 
< / property> 
<属性> 
<名称> mapreduce.jobhistory.webapp.address< / name> 
< value> localhost：8188< /值> 
< / property> 
<属性> 
< name> mapreduce.job.emit-timeline-data< / name> 
<值> true< /值> 
< / property> 
< / configuration> 
  

JPS输出：

  6022 NameNode 
 27976 NodeManager 
 27859 ResourceManager 
 6139 DataNode 
 6310 SecondaryNameNode 
 28482 ApplicationHistoryServer 
 29230 Jps 
 <如果您想通过YARN RM Web UI查看日志，那么您需要使用YARN RM Web UI来查看日志。启用日志聚合。为此，您需要在yarn-site.xml中设置以下参数： 
 
 
 < property> 
< name> yarn.log-aggregation-enable< / name> 
<值> true< /值> 
< / property> 
<属性> 
< name> yarn.nodemanager.remote-app-log-dir< / name> 
<值> / app-logs< /值> 
< / property> 
<属性> 
< name> yarn.nodemanager.remote-app-log-dir-suffix< / name> 
<值>日志< /值> 
< / property> 
  
如果您未启用日志汇总，则NM将在本地存储日志。通过上述设置，日志以/ app-logs / {用户名} / logs /的HDFS聚合。在此文件夹下，您可以找到迄今为止运行的所有应用程序的日志。再次，日志保留由配置参数yarn.log-aggregation.retain-seconds（保留聚合日志的时间）确定。
 
 
 
当MapReduce应用程序正在运行，那么您可以从YARN的Web UI访问日志。 
 
 
 
另外，在yarn-site.xml中设置以下配置参数：
 <属性> 
< name> yarn.log.server.url< / name> 
<值> http：// {job-history-hostname}：8188 / jobhistory / logs< / value> 
< / property> 
  
 
I am using the latest hadoop version 3.0.0 build from source code. I have my timeline service up and running and have configured hadoop to use that for job history also. But when I click on history in the resoucemanager UI I get the below error:-
HTTP ERROR 404

Problem accessing /jobhistory/job/job_1444395439959_0001. Reason:

    NOT_FOUND
Can someone please point out what I am missing here. Following is my yarn-site.xml:-
<configuration>

<!-- Site specific YARN configuration properties -->
<property>
  <description>The hostname of the Timeline service web application.</description>
  <name>yarn.timeline-service.hostname</name>
  <value>0.0.0.0</value>
</property>
<property>
  <description>Address for the Timeline server to start the RPC server.</description>
  <name>yarn.timeline-service.address</name>
  <value>${yarn.timeline-service.hostname}:10200</value>
</property>

<property>
  <description>The http address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.address</name>
  <value>${yarn.timeline-service.hostname}:8188</value>
</property>

<property>
  <description>The https address of the Timeline service web application.</description>
  <name>yarn.timeline-service.webapp.https.address</name>
  <value>${yarn.timeline-service.hostname}:8190</value>
</property>

<property>
  <description>Handler thread count to serve the client RPC requests.</description>
  <name>yarn.timeline-service.handler-thread-count</name>
  <value>10</value>
</property>
<property>
  <description>Indicate to ResourceManager as well as clients whether
  history-service is enabled or not. If enabled, ResourceManager starts
  recording historical data that Timelien service can consume. Similarly,
  clients can redirect to the history service when applications
  finish if this is enabled.</description>
  <name>yarn.timeline-service.generic-application-history.enabled</name>
  <value>true</value>
</property>

<property>
  <description>Store class name for history store, defaulting to file system
  store</description>
  <name>yarn.timeline-service.generic-application-history.store-class</name>
  <value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>
<property>
     <description>URI pointing to the location of the FileSystem path where the history will be persisted.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.uri</name>
     <value>/tmp/yarn/system/history</value>
</property>
<property>
     <description>T-file compression types used to compress history data.</description>
     <name>yarn.timeline-service.generic-application-history.fs-history-store.compression-type</name>
     <value>none</value>
</property>



 <property>
     <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>
</configuration>
and my mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>mapreduce.job.emit-timeline-data</name>
<value>true</value>
</property>
</configuration>
JPS output:
6022 NameNode
27976 NodeManager
27859 ResourceManager
6139 DataNode
6310 SecondaryNameNode
28482 ApplicationHistoryServer
29230 Jps

解决方案
If you want to see the logs through YARN RM web UI, then you need to enable the log aggregation. For that, you need to set the following parameters, in yarn-site.xml:
  <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
  </property>
  <property>
     <name>yarn.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
  </property>
  <property>
      <name>yarn.nodemanager.remote-app-log-dir-suffix</name>
      <value>logs</value>
  </property>
If you do not enable log aggregation, then NMs will store the logs locally. With the above settings, the logs are aggregated in HDFS at "/app-logs/{username}/logs/". Under this folder, you can find logs for all the applications run so far. Again the log retention is determined by the configuration parameter "yarn.log-aggregation.retain-seconds" (how long to retain the aggregated logs).

When the MapReduce applications are running, then you can access the logs from the YARN's web UI. Once the application is completed, the logs are served through Job History Server.

Also, set following configuration parameter in yarn-site.xml:
<property>
  <name>yarn.log.server.url</name>
  <value>http://{job-history-hostname}:8188/jobhistory/logs</value>
</property>


                        
这篇关于YARN的工作历史未到的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！