如何使用CDH4和Yarn查看我的Hadoop作业历史记录和日志？ [英] How do I view my Hadoop job history and logs using CDH4 and Yarn?

问题描述

我用Yarn下载了Hadoop的CDH4 tar，作业运行良好，但我无法弄清楚从我的工作中查看日志的位置。在MRv1中，我只是去了JobTracker网络应用程序，它有工作的历史。个人作业的日志也可以从这里访问，或者转到日志/用户日志目录。

在我的新Yarn设置中（仅在单台计算机上运行），我有日志目录，但没有日志/用户日志文件夹。

当我进入ResourceManager网页localhost：8088时，有一个所有应用程序部分，但这里没有显示任何条目。当我到JobHistory网页localhost：19888时，没有列出任何工作。

我想知道是否有配置问题。这是我的yarn-site.xml条目：

 < property> 
< name> yarn.nodemanager.local-dirs< / name> 
< value> / Users / dolan / cdh4 / _hadoop_data / yarn / local< / value> 
< / property> 
<属性> 
< name> yarn.nodemanager.log-dirs< / name> 
< value> / Users / dolan / cdh4 / _hadoop_data / yarn / logs< / value> 
< / property> 
<属性> 
<名称> mapreduce.jobhistory.address< / name> 
< value> localhost：10020< /值> 
< / property> 
<属性> 
<名称> mapreduce.jobhistory.webapp.address< / name> 
< value> localhost：19888< /值> 
< / property> 
<属性> 
< name> yarn.app.mapreduce.am.staging-dir< / name> 
<值> / user< /值> 
< / property> 
  

有关调试内容的任何提示都非常感谢。

更新：我检查了nodemanager日志目录，它是空的。另外，jobhistory目录是空的。奇怪的是，Hadoop告诉我， INFO mapreduce.Job：跟踪作业的URL：http：// localhost：8080 / ，但该地址不可用当我尝试从浏览器。

更新＃2：我注意到我运行的每个作业具有相同的作业ID（job_local_0001）。这似乎是某处的配置问题。

解决方案

听起来这里的工作是在本地运行，而不是在YARN上运行。要在YARN上运行作业，请确保在你的mapred-site.xml中有这个：

 < property> 
<名称> mapreduce.framework.name< /名称> 
<值>纱线< /值> 
< / property> 
  

一般来说，您可能会通过电子邮件发送cdh-user邮件列表。

I downloaded the CDH4 tar for Hadoop with Yarn, and jobs are running fine, but I can't figure out where to view the logs from my job. In MRv1, I simply went to the JobTracker web app, and it had the job history. Individual jobs' logs were accessible from here as well, or by going to logs/userlogs directory.

In my new Yarn setup (just running on single computer), I have the logs directory, but no logs/userlogs folder.

When I go to the ResourceManager web page, localhost:8088, there is an "All Applications" section, but no entries show here. When I go to the JobHistory web page, localhost:19888, there are no jobs listed.

I'm wondering if there is a configuration issue. Here's my yarn-site.xml entries:

<property>
  <name>yarn.nodemanager.local-dirs</name> 
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/local</value>
</property> 
<property> 
  <name>yarn.nodemanager.log-dirs</name>
  <value>/Users/dolan/cdh4/_hadoop_data/yarn/logs</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.address</name>
  <value>localhost:10020</value> 
</property>
<property> 
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>localhost:19888</value> 
</property>
<property>
  <name>yarn.app.mapreduce.am.staging-dir</name>
  <value>/user</value>
</property>

Any tips on what to debug are greatly appreciated.

Update: I've checked the nodemanager log dir, and it is empty. Additionally, the jobhistory directories are empty. One odd thing is that Hadoop tells me, INFO mapreduce.Job: The url to track the job: http://localhost:8080/, but that address isn't available when I try from browser.

Update #2: I noticed that each job I run has the same Job ID ("job_local_0001"). This seems like a configuration issue somewhere.

解决方案

It sounds like the jobs here are being run locally and not on YARN at all. To make run the jobs on YARN, make sure you have this in your mapred-site.xml:

<property>
  <name> mapreduce.framework.name</name>
  <value>yarn</value>
</property>

In general, you'll probably get a faster response on questions like this by emailing the cdh-user mailing list.

这篇关于如何使用CDH4和Yarn查看我的Hadoop作业历史记录和日志？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！