强制SGE使用多个服务器 [英] Forcing SGE to use multiple servers

问题描述

TL; DR:在安排作业时，是否有任何方法可以使SGE在服务器之间轮循，而不是在可能的情况下将所有作业分配给同一服务器?

TL;DR: Is there any way to get SGE to round-robin between servers when scheduling jobs, instead of allocating all jobs to the same server whenever it can?

详细信息:

我有一个大型的计算过程，其中包含许多较小的工作.我正在使用SGE在群集中的多个服务器之间分配工作.

I have a large compute process that consists of many smaller jobs. I'm using SGE to distribute the work across multiple servers in a cluster.

该过程在不同的时间点需要不同数量的任务(从技术上讲，它是作业的DAG).有时并行作业的数量非常大(集群中每个CPU约1个)，有时却很小(每个服务器约1个).DAG是动态的并且不是统一的，因此要确定在任何给定点有/将要进行多少并行作业并不容易.

The process requires a varying number of tasks at different points in time (technically, it is a DAG of jobs). Sometimes the number of parallel jobs is very large (~1 per CPU in the cluster), sometimes it is much smaller (~1 per server). The DAG is dynamic and not uniform so it isn't easy to tell how many parallel jobs there are/will at any given point.

作业使用大量CPU，但也执行少量的IO(尤其是在作业启动和关闭时).他们访问连接到所有计算服务器的共享NFS服务器.每个计算服务器具有较窄的连接(10Gb/s)，但是NFS服务器具有通向通信交换机的多个较宽的连接(40Gbs).不确定交换机主干的带宽是多少，但是它是一个怪物，因此应该很高.

The jobs use a lot of CPU but also do some non trivial amount of IO (especially at job startup and shutdown). They access a shared NFS server connected to all the compute servers. Each compute server has a narrower connection (10Gb/s) but the NFS server has several wide connections (40Gbs) into the communication switch. Not sure what the bandwidth of the switch backbone is, but it is a monster so it should be high.

为获得最佳性能，应在可能的情况下在不同服务器上安排作业.也就是说，如果我有20台服务器，每台服务器有20个处理器，则提交20个作业应在每个服务器上运行一个作业.提交40个作业应在每个作业上运行2个，依此类推.提交400个作业将使整个群集饱和.

For optimal performance, jobs should be scheduled across different servers when possible. That is, if I have 20 servers, each with 20 processors, submitting 20 jobs should run one job on each. Submitting 40 jobs should run 2 on each, etc. Submitting 400 jobs would saturate the whole cluster.

但是，SGE有害于将我的I/O性能降到最低.提交20个作业将在单个服务器上安排所有作业.因此，当其他19台带宽为190Gb的计算机闲置时，它们都为一个仅10Gb的网络连接而战.

However, SGE is perversely intent on minimizing my I/O performance. Submitting 20 jobs would schedule all of them on a single server. So they all fight for a single measly 10Gb network connection when 19 other machines with a bandwidth of 190Gb sit idle.

我可以强制以多种方式(使用资源，使用特殊队列，使用并行环境以及指定"-t 1-"等)在不同的服务器上执行每个作业.但是，这意味着我只能在每个服务器上期间运行一个作业.当DAG打开并产生许多作业时，这些作业将停止等待完全免费的服务器，而每台计算机的20个处理器中有19个将保持空闲状态.

I can force SGE to execute each job on a different server in several ways (using resources, using special queues, using my parallel environment and specifying '-t 1-', etc.). However, this means I will only be able to run one job per server, period. When the DAG opens up and spawns many jobs, the jobs will stall waiting for a completely free server while 19 out of the 20 processors of each machine will stay idle.

我需要的是一种告诉SGE以轮循顺序将每个作业分配给具有可用插槽的下一台服务器的方法.更好的方法是将作业分配给负载最少的服务器(最大数量的未使用插槽，或最大部分的未使用插槽，或最小数量的已使用插槽，等等).但是一个简单的死循环就可以解决问题.

What I need is a way to tell SGE to to assign each job to the next server that has an available slot in a round-robin order. A better way would be to assign the job to the least loaded server (maximal number of unused slots, or maximal fraction of unused slots, or minimal number of used slots, etc.). But a dead simple round-robin would do the trick.

与SGE的将每个作业与上一个作业在同一台服务器上运行的策略相比，这似乎是一个更明智的策略 ，这对我来说是最糟糕的策略.

This seems like a much more sensible strategy in general, compared to SGE's policy of running each job on the same server as the previous job, which is just about the worst possible strategy for my case.

我查看了SGE的配置选项，但是找不到任何修改调度策略的方法.也就是说，SGE的文档并不十分容易浏览，因此我很容易错过了一些东西.

I looked over SGE's configuration options but I couldn't find any way to modify the scheduling strategy. That said, SGE's documentation isn't exactly easy to navigate, so I could have easily missed something.

有人知道任何方法使SGE沿着这些路线将其调度策略更改为轮循或最少负荷或任何吗?

Does anyone know of any way to get SGE to change its scheduling strategy to round-robin or least-loaded or anything along these lines?

谢谢！

推荐答案

对于SGE并行环境，只需将 allocation_rule 更改为 $ round_robin ( sge_pe 文件):

Simply change allocation_rule to $round_robin for the SGE parallel environment (sge_pe file):

allocation_rule

     The allocation rule is interpreted by the  scheduler  thread
     and helps the scheduler to decide how to distribute parallel
     processes among the available machines. If, for instance,  a
     parallel environment is built for shared memory applications
     only, all parallel processes have to be assigned to a single
     machine, no matter how much suitable machines are available.
     If, however, the parallel environment  follows  the  distri-
     buted  memory  paradigm,  an  even distribution of processes
     among machines may be favorable.
     The current version of the scheduler  only  understands  the
     following allocation rules:

<int>:    An integer number fixing the number  of  processes
           per  host.  If the number is 1, all processes have
           to reside  on  different  hosts.  If  the  special
           denominator  $pe_slots  is used, the full range of
           processes as specified with the qsub(1) -pe switch
           has  to  be  allocated on a single host (no matter
           which value belonging  to  the  range  is  finally
           chosen for the job to be allocated).

$fill_up: Starting from the best  suitable  host/queue,  all
          available  slots  are allocated. Further hosts and
          queues are "filled up" as  long  as  a  job  still
          requires slots for parallel tasks.

$round_robin:
          From all suitable hosts a single slot is allocated
          until  all tasks requested by the parallel job are
          dispatched. If more tasks are requested than suit-
          able hosts are found, allocation starts again from
          the  first  host.  The  allocation  scheme   walks
          through  suitable  hosts  in a best-suitable-first
          order.

来源: http://gridscheduler.sourceforge.net/htmlman/htmlman5/sge_pe.html

这篇关于强制SGE使用多个服务器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！