如何启用公平调度程序? [英] How to enable Fair scheduler?

问题描述

我想了解Spark的FAIR调度模式的内部.事实是，根据 Spark官方文档:

I'd like to understand the internals of Spark's FAIR scheduling mode. The thing is that it seems not so fair as one would expect according to the official Spark documentation:

从Spark 0.8开始，还可以配置作业之间的公平共享.在公平共享下，Spark以循环"方式在作业之间分配任务，以便所有作业都获得大致相等的群集资源份额.这意味着在运行长作业时提交的短作业可以立即开始接收资源，并且仍然获得良好的响应时间，而无需等待长作业完成.此模式最适合多用户设置.

Starting in Spark 0.8, it is also possible to configure fair sharing between jobs. Under fair sharing, Spark assigns tasks between jobs in a "round robin" fashion, so that all jobs get a roughly equal share of cluster resources. This means that short jobs submitted while a long job is running can start receiving resources right away and still get good response times, without waiting for the long job to finish. This mode is best for multi-user settings.

似乎作业没有得到平等的处理，实际上按照fifo的顺序进行管理.

It seems like jobs are not handled equally and actually managed in fifo order.

要提供有关该主题的更多信息:

To give more information on the topic:

我在YARN上使用Spark.我使用Spark的Java API.要启用公平模式，代码为:

I am using Spark on YARN. I use the Java API of Spark. To enable the fair mode, The code is :

SparkConf conf = new SparkConf();
conf.set("spark.scheduler.mode", "FAIR");
conf.setMaster("yarn-client").setAppName("MySparkApp");
JavaSparkContext sc = new JavaSparkContext(conf);

我错过了什么吗?

推荐答案

似乎您没有设置池，并且所有作业最终都以一个 default 结束代码>池，如配置池属性中所述:

It appears that you didn't set up the pools and all your jobs end up in a single default pool as described in Configuring Pool Properties:

特定池的属性也可以通过配置文件进行修改.

Specific pools’ properties can also be modified through a configuration file.

及以后

完整的示例也可以在conf/fairscheduler.xml.template中找到.请注意，任何未在XML文件中配置的池都将简单地获得所有设置(调度模式FIFO，权重1和minShare 0)的默认值.

A full example is also available in conf/fairscheduler.xml.template. Note that any pools not configured in the XML file will simply get default values for all settings (scheduling mode FIFO, weight 1, and minShare 0).

也可能是因为您没有设置本地属性来设置池以供给定作业使用，如

It can also be that you didn't set up the local property to set up the pool to use for a given job(s) as described in Fair Scheduler Pools:

没有任何干预，新提交的作业将进入默认池，但是可以通过在提交线程的SparkContext中添加spark.scheduler.pool本地属性"来设置作业的池.

Without any intervention, newly submitted jobs go into a default pool, but jobs’ pools can be set by adding the spark.scheduler.pool "local property" to the SparkContext in the thread that’s submitting them.

这最终可能意味着您使用单个 default FIFO池，因此与没有池的FIFO相比，处于FIFO模式的一个池不会发生任何变化.

It can finally mean that you use a single default FIFO pool so one pool in FIFO mode changes nothing comparing to FIFO without pools.

只有您知道真正的答案:)

It's only you to know the real answer :)

这篇关于如何启用公平调度程序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！