“应用优先级"在纱线中 [英] "Application priority" in Yarn

问题描述

我正在使用Hadoop 2.9.0.是否可以在YARN中提交具有不同优先级的作业?根据JIRA的一些票证，似乎已经实现了应用程序优先级.

I am using Hadoop 2.9.0. Is it possible to submit jobs with different priorities in YARN? According to some JIRA tickets it seems that application priorities have now been implemented.

我尝试使用 YarnClient ，并在提交作业之前为 ApplicationSubmissionContext 设置了优先级.我还尝试使用CLI和 updateApplicationPriority .但是，似乎没有什么改变应用程序的优先级，它始终保持为0.

I tried using the YarnClient, and setting a priority to the ApplicationSubmissionContext before submitting the job. I also tried using the CLI and using updateApplicationPriority. However, nothing seems to be changing the application priority, it always remains 0.

我是否误解了YARN的ApplicationPriority概念?我看到了一些有关为队列设置优先级的文档，但是对于我的用例，我需要将所有作业都放在一个队列中.

Have I misunderstood the concept of ApplicationPriority for YARN? I saw some documentation about setting priorities to queues, but for my use case I need all jobs in one queue.

对于我对理解的任何澄清或对我可能做错事的建议，我们将不胜感激.

Will appreciate any clarification on my understanding, or suggestions about what I could be doing wrong.

谢谢.

推荐答案

是的，可以在纱线簇上设置应用程序的优先级.

Yes, it is possible to set priority of your applications on the yarn cluster.

叶子队列级别的优先级
您可以定义具有不同优先级的队列，并使用spark-submit将您的应用程序提交给具有所需优先级的特定队列.

Leaf Queue-level priority
You can define queues with different priority and use spark-submit to submit your application to the specific queue with the wanted priority.

基本上，您可以像这样在etc/hadoop/capacity-scheduler.xml中定义队列:

Basically you can define your queues in etc/hadoop/capacity-scheduler.xml like this:

<property>
  <name>yarn.scheduler.capacity.root.prod.queues</name>
  <value>prod1,prod2</value>
  <description>Production queues.</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.test.queues</name>
  <value>test1,test2</value>
  <description>Test queues.</description>
</property>

请参阅队列属性文档这里

注意:应用程序优先级仅与FIFO排序策略一起使用.默认的订购政策是FIFO.

Note: Application priority works only along with FIFO ordering policy. Default ordering policy is FIFO.

为了设置应用程序优先级，您可以将以下属性添加到同一文件中:

In order to set application priority you can add properties like this to the same file:

<property>
  <name>yarn.scheduler.capacity.root.test.default-application-priority</name>
  <value>10</value>
  <description>Test queues have low priority.</description>
</property>

<property>
  <name>yarn.scheduler.capacity.root.prod.default-application-priority</name>
  <value>90</value>
  <description>Production queues have high priority.</description>
</property>

查看有关应用程序优先级的更多信息此处

See more information about application priority here

在运行时更改应用程序优先级:
如果要在运行时更改应用程序优先级，还可以使用如下CLI:

Changing application priority at runtime:
If you want to change application priority at runtime you can also use the CLI like this:

yarn application -appId <ApplicationId> -updatePriority <Priority>

您可以共享在哪个节点上执行什么命令以及获得什么响应吗?

Can you share what command you execute on what node and what response you get?

查看更多信息这里

使用YarnClient
您没有共享您的代码，因此很难看出您是否做对了.但是可以使用YarnClient提交具有特定优先级的新应用

Using YarnClient
You did not share your code so it is difficult to see if you do it right. But it is possible to submit a new application with a specific priority using YarnClient

ApplicationClientProtocol.submitApplication(SubmitApplicationRequest)

查看更多信息 查看全文