如何在AWS EMR上更改纱线调度器配置? [英] How to change yarn scheduler configuration on aws EMR?

问题描述

与HortonWorks或Cloudera不同，AWS EMR似乎没有提供任何GUI来更改各种hadoop生态系统框架的xml配置.

Unlike HortonWorks or Cloudera, AWS EMR does not seem to give any GUI to change xml configurations of various hadoop ecosystem frameworks.

登录到我的EMR名称节点并快速进行

Logging into my EMR namenode and doing a quick

find \ -iname yarn-site.xml

我能够找到它位于/etc/hadoop/conf.empty/yarn-site.xml处，而容量调度器位于/etc/hadoop/conf.empty/capacity-scheduler.xml处.

I was able to find it to be located at /etc/hadoop/conf.empty/yarn-site.xml and capacity-scheduler to be located at /etc/hadoop/conf.empty/capacity-scheduler.xml.

但是请注意这些是如何设置为conf.empty的，我怀疑它们可能不是yarn-site和capacity-scheduler xmls的实际位置.

But note how these are under conf.empty and I suspect these might not be the actual locations for yarn-site and capacity-scheduler xmls.

我知道我可以在创建集群时更改这些配置，但是我需要知道的是如何能够在不破坏集群的情况下进行更改.

I understand that I can change these configurations while making a cluster but what I need to know is how to be able to change them without tearing apart the cluster.

我只想尝试一些调度属性，并尝试使用不同的调度程序来确定哪些可能适用于我的spark应用程序.

I just want to play around scheduling properties and such and try out different schedulers to identify what might work will with my spark applications.

提前谢谢！

推荐答案

好吧，yarn-site.xml和capacity-scheduler.xml确实位于正确的位置(/etc/hadoop/conf.empty/)上，并且在正在运行的群集上，在主节点上进行编辑并重新启动YARN RM守护程序将更改调度程序.

Well, the yarn-site.xml and capacity-scheduler.xml are indeed under correct locations (/etc/hadoop/conf.empty/) and on running cluster , editing them on master node and restarting YARN RM Daemon will change the scheduler.

启动新集群时，可以使用EMR Configurations API更改适当的值. http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

When spinning up a new cluster , you can use EMR Configurations API to change appropriate values. http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html

例如:在您的EMR配置中的capacity-scheduler and yarn-site classifications中指定适当的值，以在相应的XML文件中更改这些值.

For example : Specify appropriate values in capacity-scheduler and yarn-site classifications on your Configuration for EMR to change those values in corresponding XML files.

2019年9月4日: 使用Amazon EMR版本5.21.0和更高版本，您可以覆盖集群配置并为正在运行的集群中的每个实例组指定其他配置分类.您可以通过使用Amazon EMR控制台，AWS命令行界面(AWS CLI)或AWS开发工具包来完成此操作.

Sep 4, 2019 : With Amazon EMR version 5.21.0 and later, you can override cluster configurations and specify additional configuration classifications for each instance group in a running cluster. You do this by using the Amazon EMR console, the AWS Command Line Interface (AWS CLI), or the AWS SDK.

请参阅 https://docs.aws. amazon.com/emr/latest/ReleaseGuide/emr-configure-apps-running-cluster.html

这篇关于如何在AWS EMR上更改纱线调度器配置?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！