YARN shell命令获取正在运行的应用程序使用的容器和vcore的数量 [英] YARN shell command to get number of containers and vcores used by running applications

问题描述

hadoop job -list 似乎只显示mapreduce作业，但是我的资源管理器UI为所有正在运行的作业显示Allocated CPU VCores和Allocated Memory MB （包括诸如Spark和Tez之类的东西）。

如何通过命令行获取这些结果而不是访问UI？

$ b

解决方案

YARN支持不同类型的应用程序。 MapReduce是YARN支持的一种应用程序。如果您使用 hadoop job （不推荐使用，您应该使用 mapred job ）或 Mapreduce作业，您只能操作MapReduce作业。

要查看不同类型应用程序（mapreduce，spark等）的状态，你应该使用YARN CLI。

例如yarn application -list -appStates ALL列出所有应用程序的状态。此命令的输出包含指示应用程序类型的Application-Type列（例如，MapReduce应用程序的MAPREDUCE）。

您可以获得有关应用程序，应用程序尝试，容器等的详细信息。此处给出了2.7.1版Hadoop的命令： https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html 。但是，这些命令不会通过CLI返回分配的CPU VCores和分配的内存MB等详细信息。

$ b

yarn application -status {Application ID} 命令返回聚合资源分配根据MB-seconds和vcore-seconds

-status 返回了我的一个应用程序：

聚合资源分配：12865641 MB-seconds，1041 vcore-seconds

hadoop job -list only seems to show mapreduce jobs, but my resource manager UI shows things like "Allocated CPU VCores" and "Allocated Memory MB" for all jobs running on YARN (including things like Spark and Tez).

How can I get these results via the command line instead of going to the UI?

解决方案

YARN supports different types of applications. "MapReduce" is one type of the application supported by YARN. If you use hadoop job (which is deprecated, you should use mapred job instead) or mapred job, you can only manipulate MapReduce jobs.

To view the status of the different types of applications (mapreduce, spark etc.), you should use YARN CLI.

For e.g. "yarn application -list -appStates ALL", lists the status of all the applications. The output of this command contains a column "Application-Type" which indicates the type of the application (for e.g. MAPREDUCE for MapReduce application).

You can get the details about the application, application attempt, containers etc. The commands for 2.7.1 version of Hadoop are given here: https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnCommands.html.

But, these commands don't return the details like "Allocated CPU VCores" and "Allocated Memory MB", through CLI.

yarn application -status {Application ID} command returns "Aggregate Resource Allocation" in terms of "MB-seconds" and "vcore-seconds"

For e.g. -status for one of my applications returned:

Aggregate Resource Allocation : 12865641 MB-seconds, 1041 vcore-seconds

这篇关于YARN shell命令获取正在运行的应用程序使用的容器和vcore的数量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！