如何保持YARN的日志文件？ [英] How to keep YARN's log files?
问题描述
您无法在用户界面上看到日志，因为默认情况下，日志聚合被禁用。以下参数确定日志聚合：
yarn.log-aggregation-enable （如果禁用日志聚合，则设置为 false ）。
如果将其设置为false，则所有节点管理器都将容器日志存储在本地目录中，由以下配置参数确定：
yarn.nodemanager.log-dirs 。
例如在我的情况下，这是设置为：
< property>
< name> yarn.nodemanager.log-dirs< / name>
<值> e：\hdddata\hadoop\logs< /值>
< / property>
因此，我所有的特定应用程序的容器日志都会在文件夹e：\\中找到。 \\ b
在运行应用程序主机的节点管理器计算机中，运行\\ dpdata \ hadoop \logs\ {application-id} \ {container-id} >假设我的应用程序：application_1443377528298_0010失败。在YARNRM的用户界面中（由config参数确定： yarn.resourcemanager.webapp.address ），您可以获取有关应用程序管理器运行的节点的信息。在下图中，应用程序管理器在机器上运行120243。
如果您登录到本机并在文件夹e：\hdddata\hadoop\logs\应用程序_1443377528298_0010 \，您可以看到应用程序application_1443377528298_0010的所有容器的日志。
但是，现在如果您想通过YARN RM Web UI ，那么你需要启用日志聚合。为此，您需要在yarn-site.xml中设置以下参数：
< property>
< name> yarn.log-aggregation-enable< / name>
<值> true< /值>
< / property>
<属性>
< name> yarn.nodemanager.remote-app-log-dir< / name>
<值> / app-logs< /值>
< / property>
<属性>
< name> yarn.nodemanager.remote-app-log-dir-suffix< / name>
<值>日志< /值>
< / property>
通过以上设置，我的日志在HDFS中以/ app-logs / {username} /日志/。在此文件夹下，您可以找到迄今为止运行的所有应用程序的日志。再次，日志保留由配置参数 yarn.log-aggregation.retain-seconds （保留聚合日志的时间）确定。
当MapReduce应用程序正在运行时，您可以从YARN的Web UI访问日志。一旦应用程序完成，日志将通过作业历史记录服务器提供。
在您的情况下，如果您想在Web UI上查看日志，终止，那么你还需要开始运行MapReduce Job History服务器。要启用它，请在mapred-site.xml中设置以下配置参数：
< property>
<名称> mapreduce.jobhistory.address< / name>
< value> {job-history-hostname}：10020< /值>
< / property>
<属性>
<名称> mapreduce.jobhistory.webapp.address< / name>
<值> {job-history-hostname}：19888< /值>
< / property>
并在yarn-site.xml中设置以下配置参数：
<属性>
< name> yarn.log.server.url< / name>
<值> http：// {job-history-hostname}：19888 / jobhistory / logs< / value>
< / property>
我已经从Windows上的HDP安装中复制设置，并且这些设置适用于我。这些也应该为你工作。
有关上述每种配置的描述，请参阅以下链接：
https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn -default.xml
Suddenly, my YARN cluster has stopped working, everything I submit fails with "Exit code 1". I want to track down that problem, but as soon as an application failed, YARN deletes the log files. What is the configuration setting I have to adjust for YARN to keep these log files?
It seems your container is exiting with exit code 1.
You are unable to see the logs on the UI, because by default, the log aggregation is disabled. Following parameter determines the log aggregation: "yarn.log-aggregation-enable" (set to "false" if log aggregation is disabled).
If this is set to "false", then all the node managers store the container logs in a local directory, determined by the following configuration parameter: "yarn.nodemanager.log-dirs".
For e.g. in my case, this is set to:
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>e:\hdpdata\hadoop\logs</value>
</property>
So, all my container logs for a particular application, will be found in the folder "e:\hdpdata\hadoop\logs\ {application-id} \ {container-id}", in the Node Manager machine, where the Application Master ran.
Let's assume that my application: "application_1443377528298_0010" FAILED. In the YARNRM's UI (determined by config parameter: yarn.resourcemanager.webapp.address), you can get the information about the node, on which the Application Manager ran. In the figure below, the Application Manager ran on the machine "120243".
If you login to this machine and search in the folder "e:\hdpdata\hadoop\logs\application_1443377528298_0010\", you can see the logs for all the containers of application "application_1443377528298_0010".
But, now if you want to see the logs through YARN RM web UI, then you need to enable the log aggregation. For that, you need to set the following parameters, in yarn-site.xml:
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
With the above settings, my logs are aggregated in HDFS at "/app-logs/{username}/logs/". Under this folder, you can find logs for all the applications run so far. Again the log retention is determined by the configuration parameter "yarn.log-aggregation.retain-seconds" (how long to retain the aggregated logs).
When the MapReduce applications are running, then you can access the logs from the YARN's web UI. Once the application is completed, the logs are served through Job History Server.
In your case, if you want to see the logs on the Web UI, after the application is terminated, then you need to start running the MapReduce Job History server also. To enable it, set following configuration parameters in mapred-site.xml:
<property>
<name>mapreduce.jobhistory.address</name>
<value>{job-history-hostname}:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>{job-history-hostname}:19888</value>
</property>
And set following configuration parameter in yarn-site.xml:
<property>
<name>yarn.log.server.url</name>
<value>http://{job-history-hostname}:19888/jobhistory/logs</value>
</property>
I have replicated settings from HDP installation on Windows and these settings work for me. These should work for you also. For the description of each of the configurations mentioned above, refer the links below:
https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
这篇关于如何保持YARN的日志文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
